Pith. sign in

REVIEW 3 major objections 7 minor 108 references

SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding

T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AI scores 76.2 on drama-series narratives; humans score 95.8

desk verdict The benchmark is a solid, reusable resource; the PC-DCoT gains are confounded by inference budget and should be reframed. read the letter →

arxiv 2504.21435 v3 pith:VHXRB6IH submitted 2025-04-30 cs.CV cs.AIcs.CL

Chenkai Zhang , Yiming Lei , Zeming Liu , Haitao Leng , Shaoguo Liu , Tingting Gao , Qingjie Liu , Yunhong Wang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords seriesunderstandingvideoquestionansweringmultimodallargelanguagemodelsnarrativereasoningbenchmarkchain-of-thoughtdramalong-video
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current video-language benchmarks miss how people actually watch narrative media: stories that continue across episodes, with plots, recurring characters, and payoff. To make that gap measurable, it builds SeriesBench, 105 drama series (1,072 videos) annotated into 28 tasks and 29,196 questions spanning visuals, script, audio, post-production, and comprehension. On this test, strong video models score 49.8 to 62.8 percent accuracy while humans score 95.8, a much larger gap than single-video benchmarks report. The paper then proposes PC-DCoT, an inference-time framework that extracts plot events and character timelines, retrieves the relevant frames for each, and merges the two chains; it lifts four top models by 12.9 to 14.1 points, to 73.3-76.2, still short of human performance. A sympathetic reading: if correct, series-level narrative understanding is a distinct capability current MLLMs lack, and structured retrieval plus dual-chain reasoning partially supplies it.

What carries the argument

The load-bearing object is PC-DCoT, the Plot & Character Dual Chain of Thought. It works in three stages: (1) a question-conditioned MLLM extracts named events and characters from the video frames; (2) a CLIP-style retriever, finetuned on 6,046 annotated image-text pairs from SeriesBench, selects frames for each event and each character by thresholded relevance, aggregates neighboring frames within a window, and the MLLM writes descriptions into a Plot Event Chain with time intervals and Character Temporal Chains with per-character timestamps; (3) the chains are merged where a character's appearance interval intersects an event interval, and the MLLM produces a consolidated narrative text from which it answers. The benchmark's annotation pipeline is the second mechanism: long-span narrative annotation produces declarative statements, and full-information transformation converts these statements, together with subtitles, themes, and character introductions, into questions across the five task dimensions.

What would settle it

Run the four base models and their PC-DCoT versions on the same questions while keeping the input identical: same number of video frames, same subtitle text, and same character sheet. If the 12.9 to 14.1 point advantage shrinks to near zero, the dual-chain structure itself is not the cause; if it survives, the mechanism is validated.

Watch

Extended reading notes

Core claim

SeriesBench is introduced as the first evaluation built around narrative-driven series rather than isolated clips. The dataset contains 105 curated series and 1,072 videos, with subtitles, theme descriptions, and character backgrounds; annotators label key events and character actions across long spans as declarative statements, and those statements are machine-transformed with GPT-4o into true/false, multiple-choice, and open-ended questions. The central experimental discovery is that ten state-of-the-art MLLMs, including GPT-4o, perform at roughly 50-63 percent accuracy, far below the human score of 95.8, with the worst deficits in plot reasoning, character motivation, and multi-episode engagement. Applying PC-DCoT to the four best models yields consistent gains of 12.9-14.1 points (best overall 76.2 with GPT-4o), and ablations show that removing either the plot-event chain or the character-temporal chain degrades performance. The paper's own conclusion is that current models can describe actions and objects but cannot reconstruct narrative structure across episodes, and PC-DCoT is a partial remedy rather than a full solution.

Load-bearing premise

The claim that PC-DCoT improves narrative reasoning assumes its gains come from the two-chain retrieval and alignment design, not from the extra frames, subtitles, and character descriptions it feeds the model; the experiments never hold the amount of input constant between baseline and method.

Editorial extensions

If this is right

  • A model can score well on standard clip-level video QA yet sit about 33 points below humans on series-level narrative tasks, so leaderboards built on standalone-video benchmarks do not certify narrative understanding.
  • Subtitles are the most informative input: questions plus subtitles outperform questions plus frames for every tested model, and adding theme and character profiles helps further.
  • PC-DCoT consistently improves four different MLLM architectures by 12.9 to 14.1 points at inference time, suggesting the benefit transfers across model families rather than being tuned to one model.
  • Multi-episode context is not automatically helpful: earlier episodes help only some models on plot and empathy tasks, while later episodes tend to hurt all three open-source models tested.
  • Neither chain alone is enough: ablations dropping either the Plot-Event chain or the Character-Temporal chain reduce accuracy across all four models, so temporal alignment of events and characters is doing real work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • SeriesBench may partly measure text comprehension: because subtitles alone beat frames alone and many tasks follow dialogue, scores conflate narrative video understanding with reading a transcript; a vision-only variant with subtitles removed as text would separate the two.
  • PC-DCoT's gains are not isolated from increased input: baselines use fixed 32-64 sampled frames while PC-DCoT retrieves all high-relevance event and character frames and adds subtitles plus character background; holding frame count and text tokens constant would determine whether the dual-chain structure itself, rather than simply more context, produces the improvement.
  • The retriever is trained on SeriesBench's own annotation distribution, so applying PC-DCoT to other long-form video domains would test whether the event-and-character retrieval step is portable or overfitted to these drama series.
  • The declarative event statements themselves could serve as supervision for narrative-state tracking: a follow-up that fine-tunes a model on the annotations and measures transfer to unseen series would test whether the benchmark's content, not just its questions, improves narrative understanding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces SeriesBench, a benchmark for evaluating MLLMs on narrative-driven drama series, containing 105 curated series, 1,072 videos, 29,196 tasks, and a five-dimension taxonomy covering visuals, script, audio, augmentation, and comprehension with 28 subtasks. The benchmark is built from manual long-span narrative annotations by 32 annotators followed by a GPT-4o 'full-information transformation' into multiple-choice, true/false, and open-ended questions. The authors evaluate ten video/audio MLLMs and report large gaps between model and human performance (best model 76.2 vs. human 95.8). They also propose PC-DCoT, a Plot Event Chain and Character Temporal Chain retrieval-and-reasoning framework, and report accuracy gains of +12.9 to +14.1 points on four models, concluding that PC-DCoT consistently achieves state-of-the-art performance on SeriesBench.

Significance. The benchmark contribution is timely and useful: SeriesBench is, to my knowledge, the first series-level narrative understanding benchmark with multi-modal task dimensions, and it is publicly released. The manual annotation workflow with reviewer-based quality control (96% pass on a random 500-sample check) and the breadth of evaluated models (10 MLLMs including audio-video and commercial systems) make the main empirical finding credible: current MLLMs are far below human performance on narrative-driven series understanding. The PC-DCoT idea is reasonable and the paper includes useful ablations of input modalities, but the method-specific claims are currently overstated. The central contribution that survives scrutiny is the benchmark and the finding that narrative understanding across episodes remains unsolved; the framework's superiority over simply adding more context is not yet established.

major comments (3)
  1. [Section 5.2 and Table 3] The claim that 'PC-DCoT consistently achieves SOTA performance across all tasks' is not supported by the reported numbers. Table 10 shows several fine-grained tasks where a model without PC-DCoT is the best performer, e.g., Character Resonance: LLaVA-Video 59.1 vs. the best PC-DCoT model at 47.6; Future Predictions: Qwen2-VL 70.6 vs. the best PC-DCoT model at 68.8; Time and Location: Qwen2-VL 78.9 vs. the best PC-DCoT model at 75.0. In addition, two of the four PC-DCoT models are worse than their base counterparts on all open-ended metrics in Table 3 (InternVL2†: BLEU-2 8.26 vs. 8.43, METEOR 20.71 vs. 21.22, BERTScore 67.85 vs. 68.40; Qwen2-VL†: BLEU-2 8.93 vs. 11.41, METEOR 23.41 vs. 27.97, BERTScore 68.08 vs. 70.71). The conclusion should be restricted to closed-form accuracy on the aggregate benchmark, and the open-ended regression should be discussed explicitly.
  2. [Section 5.2 and Appendix E.2] The reported PC-DCoT gains are not controlled for inference budget or input modalities. The baselines use 32–128 uniformly sampled frames (Table 3), whereas PC-DCoT, per Eqs. (3)–(10), adds event- and character-retrieved frames using thresholds θ_e and θ_c, expands them by window δ, and prepends the Plot Event Chain, Character Temporal Chain, and a synthesized narrative before the question. Table 5 shows that simply adding subtitles (Q,S) or thematic-character text (Q,F,S,TC) already yields large accuracy gains (e.g., InternVL2: +14.1 and +18.1 overall), so the extra context alone could explain much of the improvement. The Table 6 ablation removes an entire chain, which also removes a large share of retrieved frames and their descriptions, so it does not isolate the dual-chain mechanism. The paper should hold the frame budget and text budget approximately constant, or ablate with identical retrieved frames but without the chain text, before attributing the gains to narrative-reasoning design.
  3. [Section 3.2.2] The GPT-4o full-information transformation is not validated for question quality or answer leakage. The 29,196 tasks are generated from manual declarative statements with 'related video information crafted as distractors,' but no human evaluation of the generated questions, no question-only baseline, and no leakage analysis is reported. The 500-annotation quality check described in Section 3.2.1 applies to the annotations, not to the automatically generated questions. Since GPT-4o is also one of the evaluated models, the benchmark should demonstrate that correct answers cannot be recovered from question text alone or from the manual statements alone, and should report human agreement on a sample of generated tasks.
minor comments (7)
  1. [Section 5.3 (RQ1) and Table 4] The text is internally inconsistent: it first says earlier episodes improve performance and then says only Qwen2-VL improves with earlier episodes, while Table 4 shows InternVL2 with Prev1 (65.2) above its no-episode baseline (64.0). The narrative and the table need to be aligned.
  2. [Table 4] The no-episode 'Total' row is 64.0 for all three models; if this is a coincidence it is surprising, and if it is a subset score the composition of 'Total' should be stated.
  3. [Appendix F.2, Table 9] The Qwen2-VL rows list 'Prev 2' twice and never list 'Prev 1'; the second row should presumably be 'Prev 1'.
  4. [Section 4 and Appendix E.2] The symbol C_j is reused for characters in Eq. (2) and for the retrieved character-frame set in Eq. (6), which makes the formalization confusing; distinct notation should be used.
  5. [Tables 3–6] No error bars, confidence intervals, or significance tests are reported for any result; given subtask sample sizes as small as 177–229 (Fig. 3), single-run accuracies are insufficient to support fine-grained ranking claims.
  6. [Section 5.2] The text says PC-DCoT was applied to 'GPT-4' while Table 3 and Appendix F.1 report GPT-4o; the model name should be made consistent.
  7. [Appendix C] The human evaluation description says each session randomly samples 10 videos from the test set, but Table 3 reports human performance across all task dimensions; the number of human-answered questions and annotators should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SeriesBench is an external benchmark and PC-DCoT is evaluated on a held-out split, with no fitted input renamed as a prediction or load-bearing self-citation.

full rationale

I walked the paper's derivation chain: SeriesBench is built from manually annotated narrative events and characters, with questions generated from those annotations via GPT-4o; the benchmark is an external evaluation resource rather than a quantity derived from the models being evaluated. PC-DCoT's retriever is trained on the SeriesBench training split and tested on the held-out test split, so its reported gains are not forced by construction. The paper does not fit a parameter to the test set and then call the fit a prediction, and the dual-chain ablation in Table 6 compares against PC-DCoT variants rather than renaming the fitted input. No uniqueness theorem or other load-bearing claim is imported from the authors' prior work; the only overlapping-author citation (Kwaichat) appears in related work and is not used to justify a central premise. The strongest weakness in the paper—that PC-DCoT's gains are not controlled for total inference budget because baselines use fixed uniform sampling while PC-DCoT adds retrieved frames, subtitles, and chain text—is a legitimate experimental-control concern, not a circularity in the logical or definitional sense. Under the stated rules, no step reduces by definition or by self-citation to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the quality of the Kuaishou micro-drama corpus, the annotation-to-question transformation, and the retrieval-based chain-of-thought inference. No new physical or mathematical entities are introduced; the main unverified inputs are the GPT-4o transformation and the CLIP retriever thresholds.

free parameters (3)
  • Event relevance threshold theta_e = not reported
    Filters frames matched to each extracted event in Eq. (3); value chosen by hand and not swept.
  • Character relevance threshold theta_c = not reported
    Filters frames matched to each character in Eq. (6); value not reported.
  • Temporal neighborhood window delta = not reported
    Adds neighboring frames to each event segment in Eq. (4); controls how much context enters the Plot Event Chain.
assumptions (3)
  • domain assumption CN-CLIPViT-H/14 similarity scores reliably retrieve event- and character-relevant frames.
    Invoked in Appendix E.1 and E.2; no retrieval precision, recall, or human verification is reported.
  • ad hoc to paper GPT-4o full-information transformation preserves ground truth when generating 29,196 questions from manual statements.
    Section 3.2.2 uses GPT-4o to convert annotations into tasks; validity of this conversion is not separately validated.
  • domain assumption MLLMs' extracted events and characters, and their chain descriptions, are accurate enough for PC-DCoT reasoning.
    Appendix E.3 prompts assume reliable extraction; no error analysis of the extraction stage is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding." pith.science (2026). https://pith.science/paper/VHXRB6IH

@misc{pith2026250421435,
  author       = {Pith},
  title        = {Pith review of: SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHXRB6IH}},
  note         = {Machine review of arXiv:2504.21435}
}
read the original abstract

With the rapid development of Multi-modal Large Language Models (MLLMs), an increasing number of benchmarks have been established to evaluate the video understanding capabilities of these models. However, these benchmarks focus on standalone videos and mainly assess "visual elements" like human actions and object states. In reality, contemporary videos often encompass complex and continuous narratives, typically presented as a series. To address this challenge, we propose SeriesBench, a benchmark consisting of 105 carefully curated narrative-driven series, covering 28 specialized tasks that require deep narrative understanding. Specifically, we first select a diverse set of drama series spanning various genres. Then, we introduce a novel long-span narrative annotation method, combined with a full-information transformation approach to convert manual annotations into diverse task formats. To further enhance model capacity for detailed analysis of plot structures and character relationships within series, we propose a novel narrative reasoning framework, PC-DCoT. Extensive results on SeriesBench indicate that existing MLLMs still face significant challenges in understanding narrative-driven series, while PC-DCoT enables these MLLMs to achieve performance improvements. Overall, our SeriesBench and PC-DCoT highlight the critical necessity of advancing model capabilities to understand narrative-driven series, guiding the future development of MLLMs. SeriesBench is publicly available at https://github.com/zackhxn/SeriesBench-CVPR2025.

Figures

Figures reproduced from arXiv: 2504.21435 by the authors.

Figure 1
Figure 1. An example from SeriesBench. The task involves mul￾tiple events and characters spanning a long time period in a video. sual understanding [4, 10, 17, 34, 35, 41, 42, 52]. Building upon these advancements, a range of benchmarks [8, 21, 25, 38, 46, 49] have been established to evaluate the visual￾language capabilities of these models, primarily through question-answering (QA) tasks based on static images. Al￾though th… view at source ↗
Figure 2
Figure 2. Illustration of PC-DCoT. The process begins by extracting relevant events and characters from the input question and the original video. Subsequently, a Plot Event Chain and a Character Temporal Chain are constructed independently. Finally, these chains are merged, enabling a reasoning process that generates an answer to the posed question based on the integrated dual-chain framework. false questions, multiple-choic… view at source ↗
Figure 3
Figure 3. Task Dimension in SeriesBench. Detailed sample count for each task in SeriesBench. Atmosphere, assessing how background music influences the emotional atmosphere of the scene. 3.3 Sound Ef￾fects: The model should understand the role of sound ef￾fects: (3.3.1) Impact, analyzing how sound effects enhance a scene’s emotional or narrative aspects. 4. Augmentation. Modern videos are no longer purely composed of footage; … view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Video Categories in SeriesBench. The videos in Se￾riesBench are categorized into two main types: thematic videos and series videos. These encompass 11 of the most popular video themes, including Urban Life, Romance, Fantasy, Counterattack, Family, Ancient Style, Campus…
Figure 5
Figure 5. Figure 5: Illustration of the web-based annotation interface. Displaying parsed video information for reference, a hierarchical video index for navigation, and an annotation area for detailed content analysis and categorization. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Example of the annotation process. Showing the selection of a video series, sequential annotation of episodes, and detailed documentation of key events, timelines, and character interactions, culminating in task categorization and a synthesized declarative sum￾mary to …
Figure 8
Figure 8. Figure 8: The prompt template adopted for Evaluation on SeriesBench. tion metrics to assess model outputs from both lexical and semantic perspectives [6, 23, 40]. BLEU-2 evaluates the bi-gram (two-word phrase) overlap between generated and reference texts, reflecting the degree …
Figure 9
Figure 9. Figure 9: The prompt template adopted for Multi-Episode series tasks. For narrative-driven videos, understanding the plot and characters is essential, as they serve as the key drivers of the storyline. Our method begins by utilizing the Mul￾timodal Large Language Model (MLLM) to…
Figure 10
Figure 10. Figure 10: Prompt for Target Extraction. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Prompt for Plot Event Chain. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Prompt for Character Temporal Chain. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Constructing the PC-DCoT. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: More qualitative cases in SeriesBench. The performance of MLLM can be improved by using the PC-DCoT framework. We use Green to indicate correct and Red to indicate incorrect. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Additional examples of open-ended responses using PC-DCoT. MLLMs are capable of conducting more detailed analyses of events, relationships, and individuals, demonstrating a deeper understanding of narrative-driven series. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

108 extracted references · 63 canonical work pages

  1. [1]

    Touchstone: Evaluating vision-language models by language models, 2023

    Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jin- gren Zhou. Touchstone: Evaluating vision-language models by language models, 2023. 2

    2023

  2. [2]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 6, 15

    2005

  3. [3]

    Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering

    Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering. arXiv preprint arXiv:2311.14906, 2023. 2, 4

    arXiv 2023

  4. [4]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms. arXiv preprint arXiv:2406.07476, 2024. 1, 3, 5, 6, 7, 21

    arXiv 2024

  5. [5]

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, and Yuki M. Asano. Tvbench: Redesign- ing video-language evaluation, 2024. 1, 4

    2024

  6. [6]

    Llm evaluation metrics ex- plained, 2024

    Data Science in Your Pocket. Llm evaluation metrics ex- plained, 2024. Accessed: 2024-11-22. 18

    2024

  7. [7]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing. arXiv preprint arXiv:2406.14515, 2024. 4

    arXiv 2024

  8. [8]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1, 2

    arXiv 2023

Show all 108 references
  1. [9]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 1, 2...

    2024 arXiv

  2. [10]

    Vita: Towards open-source interactive omni multimodal llm, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, YifanZhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He andRongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal llm, 2024. 1, 6, 7, 21

    2024

  3. [11]

    After digital story- telling: Video composing in the new media age

    Megan Fulwiler and Kim Middleton. After digital story- telling: Video composing in the new media age. Computers and Composition, 29(1):39–50, 2012. 2, 12

    2012

  4. [12]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  5. [13]

    TGIF-QA: toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QA: toward spatio-temporal reasoning in visual question answering. In CVPR, 2017. 4

    2017

  6. [14]

    Video Question Answering with Spatio-Temporal Reasoning

    Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Video Question Answering with Spatio-Temporal Reasoning. IJCV, 2019. 2

    2019

  7. [15]

    Seed-bench-2-plus: Benchmarking multi- modal large language models with text-rich visual compre- hension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multi- modal large language models with text-rich visual compre- hension. arXiv preprint arXiv:2404.16790, 2024. 2

    2024 arXiv

  8. [16]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 6, 7, 21

    2024

  9. [17]

    Aria: An open multimodal native mixture-of- experts model, 2024

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of- experts model, 2024. 1, 6, 7, 21

    2024

  10. [18]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. 2

    2024

  11. [19]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  12. [20]

    Vlm-eval: A general evaluation on video large language models

    Shuailin Li, Yuang Zhang, Yucheng Zhao, Qiuyue Wang, Fan Jia, Yingfei Liu, and Tiancai Wang. Vlm-eval: A general evaluation on video large language models. arXiv preprint arXiv:2311.11865, 2023. 2

    2023 arXiv

  13. [21]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. ArXiv, abs/2305.10355, 2023. 1

    2023 arXiv

  14. [22]

    Videovista: A versatile bench- mark for video understanding and reasoning, 2024

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile bench- mark for video understanding and reasoning, 2024. 1, 2, 4, 6, 8

    2024

  15. [23]

    Mateval: A multi-agent discussion framework for advancing open-ended text evaluation, 2024

    Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, and Dehai Min. Mateval: A multi-agent discussion framework for advancing open-ended text evaluation, 2024. 18

    2024

  16. [24]

    Video-llava: Learning united visual repre- sentation by alignment before projection, 2024

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection, 2024. 2 9

    2024

  17. [25]

    Mm- bench: Is your multi-modal model an all-around player? ArXiv, abs/2307.06281, 2023

    Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mm- bench: Is your multi-modal model an all-around player? ArXiv, abs/2307.06281, 2023. 1

    2023 arXiv

  18. [26]

    Temp- compass: Do video llms really understand videos? ArXiv preprint, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Temp- compass: Do video llms really understand videos? ArXiv preprint, 2024. 4

    2024

  19. [27]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv: 2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv: 2403.00476, 2024. 1, 2

    2024 arXiv

  20. [28]

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Chang Wen Chen, and Ying Shan. E.t. bench: Towards open-ended event-level video-language understanding. In Neural Infor- mation Processing Systems (NeurIPS), 2024. 1

    2024

  21. [29]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 15

    arXiv

  22. [30]

    Visual postproduction in participatory video- making processes

    Monica Mak. Visual postproduction in participatory video- making processes. Handbook of participatory video , pages 194–207, 2012. 2, 12

    2012

  23. [31]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 2

    2023

  24. [32]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, 2024. 4

    2024

  25. [33]

    Video-bench: A com- prehensive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A com- prehensive benchmark and toolkit for evaluating video-based large language models. ArXiv preprint, 2023. 4

    2023

  26. [34]

    Gpt-4o mini: Advancing cost-efficient intelligence,

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence,

  27. [35]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. Accessed: 2024-10-31. 1, 5, 6, 7, 21

    2024

  28. [36]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318,

  29. [37]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 5

    2021

  30. [38]

    Kwaichat: A large-scale video-driven multilingual mixed-type dialogue corpus.arXiv preprint arXiv:2503.06899, 2025

    Xiaoming Shi, Zeming Liu, Yiming Lei, Chenkai Zhang, Haitao Leng, Chuan Wang, Qingjie Liu, Wanxiang Che, Shaoguo Liu, Size Li, et al. Kwaichat: A large-scale video-driven multilingual mixed-type dialogue corpus.arXiv preprint arXiv:2503.06899, 2025. 1

    2025 arXiv

  31. [39]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024

    InternVL Team. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. Accessed: 2024- 10-31. 3, 5, 6, 7, 21

    2024

  32. [40]

    7 ways to monitor large language model behavior, 2024

    Towards Data Science. 7 ways to monitor large language model behavior, 2024. Accessed: 2024-11-22. 18

    2024

  33. [41]

    A survey on data synthesis and augmentation for large language models

    Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shi- wei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, et al. A survey on data synthesis and augmentation for large language models. arXiv preprint arXiv:2410.12896, 2024. 1

    arXiv

  34. [42]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  35. [43]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu and Shoubin Yu. Star: A benchmark for situated reasoning in real-world videos. In NeurIPS, 2024. 4

    2024

  36. [44]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 9777–9786, 2021. 4

    2021

  37. [45]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In ACM Multimedia, 2017. 2, 4

    2017

  38. [46]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. 1, 2

    2023 arXiv

  39. [47]

    Chinese clip: Contrastive vision-language pretraining in chinese

    An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335, 2022. 18

    2022 arXiv

  40. [48]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 3, 6, 7, 21

    2024 arXiv

  41. [49]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023. 1, 2

    2023

  42. [50]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019. 4

    2019

  43. [51]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text gen- eration with bert. In International Conference on Learning Representations, 2020. 6, 15

    2020

  44. [52]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 1, 6, 7, 21

    2024

  45. [53]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng 10 Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  46. [55]

    More Details of SeriesBench A.1

    2 11 A. More Details of SeriesBench A.1. Task Dimension Definitions Modern videos have diverse and intricate elements, includ- ing visuals, scripts, audio, and post-production enhance- ments. To facilitate a more comprehensive evaluation of large models that aligns with the di...

  47. [56]

    Visuals focus on understanding and analyz- ing visual content in the video

    Visuals. Visuals focus on understanding and analyz- ing visual content in the video. 1.1 Figures: The model needs to analyze the figures appearing in the video: (1.1.1) Actions, analyzing what the figures are doing; (1.1.2) Interactions, understanding interactions between figu...

  48. [57]

    Script assesses the model’s understanding of the background, plot, and characters within the video

    Script. Script assesses the model’s understanding of the background, plot, and characters within the video. 2.1 Background: The model needs to understand the setting of the story: (2.1.1) World-building, analyzing the overarch- ing background framework constructed in the video...

  49. [58]

    Audio elements assess the model’s under- standing of sound-based information in the video, includ- ing dialogues, background music, and sound effects

    Audio. Audio elements assess the model’s under- standing of sound-based information in the video, includ- ing dialogues, background music, and sound effects. 3.1 Dialogue: The model needs to understand and analyze the details of dialogues: (3.1.1) Dialogue Attribution, matchin...

  50. [59]

    Modern videos are no longer purely composed of footage; many post-production elements, such as subtitles, labels, and special effects, are added

    Augmentation. Modern videos are no longer purely composed of footage; many post-production elements, such as subtitles, labels, and special effects, are added. The model needs to understand and utilize this information to enhance video comprehension. 4.1 Subtitles: The model n...

  51. [60]

    keep” button or reset the interface for the next task using the “clear

    Comprehension. Comprehension tasks evaluate the model’s overall grasp of the video. 5.1 Engagement: The model needs to infer viewers’ interest in the plot’s devel- opment: (5.1.1) Future predictions, predicting future de- velopments based on current information; (5.1.2) Curren...

  52. [61]

    Identify all characters explicitly mentioned in the question

  53. [62]

    Include any implicitly or indirectly referenced characters related to the question

  54. [63]

    Provide names and describe their involvement in the events

  55. [64]

    Clearly describe the core events related to the question

  56. [65]

    Include secondary events from the video frames that provide context or additional clues

  57. [66]

    List events in logical or chronological order, maintaining a clear cause-and-effect chain

  58. [67]

    Background Information (Optional):

  59. [68]

    Characters

    If the question or video frames include additional background information, such as scene descriptions or timestamps, integrate them as part of the event details. Output Format Use the following JSON structure for the output: • Characters: List all relevant characters, ordered ...

  60. [69]

    Extract all characters mentioned or implied in the question

  61. [70]

    Ensure events cover all necessary details based on the video frames and the question

  62. [71]

    Number events in logical or chronological order to maintain clarity

  63. [72]

    Exclude Irrelevant Information:

  64. [73]

    Focus only on details directly relevant to the question

  65. [74]

    Figure 10

    Describe events as thoroughly as possible, including actions, background, and potential intentions. Figure 10. Prompt for Target Extraction. 24 Prompt for Plot Event Chain ( ) Based on the input event descriptions, time ranges, and video frames, generate detailed event informa...

  66. [75]

    Generate a clear and concise description for each event, summarizing the core content of the event

  67. [76]

    Ensure the description aligns with the input and includes all critical information

  68. [77]

    Each event must include a precise time range in the format MM:SS - MM:SS

  69. [78]

    Detailed Information:

  70. [79]

    No dialogue

    For each event, generate the following fields: • action: Describe the key actions taking place during the event. • scene: Provide a detailed depiction of the environment, location, or behavior of characters. • dialogue: Include any dialogue in the event. If no dialogue exists,...

  71. [80]

    These fields should be presented in a list, allowing for multiple actions or scenes per event

  72. [81]

    The result must be output in standard JSON format

  73. [82]

    Event1": {

    Each event in the JSON output must include: • time: The time range of the event in the format MM:SS - MM:SS. • description: A brief summary of the event. • details: Detailed information about the event, including actions, scenes, and dialogues. Output Format Use the following ...

  74. [83]

    Character Identification:

  75. [84]

    Extract character names and related information from the input, including portrait descriptions (e.g., appearance, clothing, etc.)

  76. [85]

    Identify all timestamps and frames where each character appears

  77. [86]

    Time Period Segmentation:

  78. [87]

    Organize the character's appearance data by grouping discrete or continuous frames and timestamps into specific time periods (defined by start and end frames)

  79. [88]

    Ensure no overlap or omission between time periods, covering all relevant frames

  80. [89]

    Action and Scene Descriptions:

  81. [90]

    For each time period, provide a detailed description of the character’s actions and behaviors

  82. [91]

    Action descriptions must be inferred from the frame data and context, providing clear and specific details about the character’s activities

  83. [92]

    Dialogue Annotation:

  84. [93]

    No dialogue

    Record the character‘s dialogue for each time period, or state "No dialogue" if absent

  85. [94]

    Output Structured Data:

  86. [95]

    Output each character’s timeline in JSON format, ensuring clarity and completeness

  87. [96]

    Character

    The output must include the character’s name, time periods with start and end frames, action descriptions, and dialogue. Output Format Use the following JSON structure for the output: { "Character": "[Character Name]", "Time Periods": [ { "time": "[Start time] - [End time]", "...

  88. [97]

    Expand Event Chains:

  89. [98]

    Develop each event with a detailed account of its cause, progression, and outcome

  90. [99]

    Maintain logical or causal relationships between events to create a cohesive storyline

  91. [100]

    Describe Characters and Interactions:

  92. [101]

    Include detailed actions, expressions, emotions, and attitudes of all involved characters

  93. [102]

    Highlight interactions between characters, including verbal exchanges, gestures, and implicit behaviors

  94. [103]

    Incorporate Dialogue:

  95. [104]

    Recreate natural dialogue relevant to the scene, showcasing the tone, intent, and personality of the characters

  96. [105]

    Ensure dialogue drives the narrative forward or reveals additional character traits or motives

  97. [106]

    Add Contextual Details:

  98. [107]

    Use descriptive elements to recreate the environment, mood, and setting of the events

  99. [108]

    Event1": {

    Clarify the time and place of each event using timestamps, spatial markers, or visual cues provided in the input. Output Format Use the following JSON structure for the output: { "Event1": { "time": "[Start time] - [End time]", "description": "[Detailed description of the even...

  100. [2024]

    1, 6, 7, 21

    Accessed: 2024-10-31. 1, 6, 7, 21

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.