Pith. sign in

REVIEW 4 major objections 5 minor 73 references

Understanding Long Videos via LLM-Powered Entity Relation Graphs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GraphVideoAgent claims that replacing sequential video memory with a dynamic entity-relation graph lets an LLM agent answer long-video questions using about 8 frames per video.

desk verdict A plausible graph-memory idea for long-video QA, but the reported gains over VideoAgent rest on single runs without variance, so the paper needs code and statistics before the claim is trustworthy. read the letter →

arxiv 2501.15953 v1 pith:LC774AYP submitted 2025-01-27 cs.IR cs.CV

Meng Chu , Yicong Li , Tat-Seng Chua This is my paper · ORCID
classification cs.IRcs.CV
keywords LLMagentlong-formvideounderstandingentityrelationgraphquestionansweringadaptiveframeselectiontemporalreasoningmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GraphVideoAgent, a system that answers questions about long videos by maintaining a dynamic graph of entities and their relations as it samples frames. The central claim is that this graph, rather than a sequential memory of frames, is what lets an LLM agent find the decisive moments in a long video. If the claim is right, structured semantic memory improves both accuracy and efficiency: the paper reports 56.3% on EgoSchema and 73.3% on NExT-QA while using only 8.2 and 8.1 frames per video on average. A sympathetic reading is that the graph's temporal tracking of entities and relations, not a stronger vision backbone, drives the improvement.

What carries the argument

The load-bearing object is a multi-level dynamic entity-relation graph $G=(V,E)$, whose nodes are entity profiles $(F_i, x_i, c_i, s_i)$ tracking appearance frames, visual features, caption descriptions, and state-change histories, and whose edges are typed relations (spatial, interaction, action) extracted via dependency parsing. Temporal coherence is maintained by the score $T(e,f)=\alpha S(e,f)+(1-\alpha)R(e,f)$, which combines entity state consistency and relation persistence, and the graph is updated across frames by $G_{t+1}=U(G_t,F_{t+1},R_{t+1})$. The LLM agent reads this graph, scores candidate frames with $S(f)=0.5\,s_{\text{graph}}(f)+0.3\,s_{\text{visual}}(f)+0.2\,s_{\text{temporal}}(f)$, and stops retrieval when its self-reflection reaches confidence level 3.

What would settle it

Run GraphVideoAgent on EgoSchema while logging the self-reported confidence and the set of frames seen at each iteration, then compare answers produced at level 3 with answers produced when the agent is forced to retrieve additional rounds; if level-3 answers are no more accurate than forced-continue answers, the confidence-based stopping rule is not the mechanism behind the reported accuracy.

Watch

Extended reading notes

Core claim

GraphVideoAgent is a video question-answering pipeline in which a vision-language model captions uniformly sampled frames, NLP tools extract entities and relations into a dynamic graph, and an LLM agent iteratively selects additional frames until a self-reported confidence level of 3 is reached. The paper's central claim is that this graph memory, not the raw frame sequence, is what lets the agent answer causal and temporal questions correctly with very few frames. On EgoSchema it reports 56.3% on the full test set (62.7% on the public subset) using 8.2 frames per video on average, and on NExT-QA it reports 73.3% overall using 8.1 frames, with gains in every question category over the VideoAgent baseline. The conclusion states that these results validate the effectiveness of structured semantic memory in video understanding tasks.

Load-bearing premise

The system trusts the LLM's self-reported level-3 confidence as proof that the already-gathered frames contain the answer, but the paper provides no calibration data showing that level-3 self-reports track actual correctness.

Editorial extensions

If this is right

  • If the central claim holds, graph-guided retrieval reaches sufficient confidence after about 8 frames, so per-video inference cost stays near constant even on hour-long inputs.
  • The EgoSchema full-set result of 56.3% with 8.2 frames would put the method above the reported VideoAgent baseline (54.1%) and roughly level with proprietary models that process many more frames.
  • On NExT-QA, the method reports gains on every question type, with the largest absolute gain over VideoAgent on causal questions (74.6% versus 72.7%).
  • The ablations imply that the multi-level graph structure, not the LLM backbone alone, accounts for the largest part of the measured improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the stopping rule would log confidence and correctness per query; if level-3 stops are not more accurate than forced extra retrievals, the reported frame efficiency would not reflect a reliable signal.
  • Because the graph is built from captions, swapping the captioning model while freezing the graph machinery would separate the contribution of richer text input from the contribution of graph structure.
  • The graph is built per question in this design; reusing it across multiple questions on the same video is an untested extension that could amortize construction cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GraphVideoAgent, an LLM-driven agent for long-form video question answering that maintains a dynamic entity-relation graph built from sampled frame captions. The graph tracks entities, their spatial/interaction/action relations, and state changes, and the agent uses it to guide iterative frame retrieval and answer generation with a self-reported confidence stop rule. The authors report state-of-the-art accuracy on EgoSchema (56.3% full, 62.7% subset) and NExT-QA (73.3%) while using only about 8.1–8.2 frames per video on average, and they present ablations on LLM choice, graph components, and entity scale, plus case studies.

Significance. If the reported results are reliable, the core idea—replacing sequential memory with an explicit, dynamically updated entity-relation graph for long-video QA—is valuable and could improve both accuracy and frame efficiency. The paper provides a concrete instantiation of this idea, with a working system and comparisons against strong baselines, and the ablations are directionally consistent with the graph contributing to performance. However, the evidence is not yet statistically robust: the gains over the primary baseline are small (+2.2 and +2.0 points), no variance or significance tests are reported, and key algorithmic components (the graph update function and the frame-scoring weights) are under-specified. The entity-scale analysis in particular rests on only 50 questions from an undescribed dataset, so the scalability claim is not supported. The manuscript also contains template placeholders and citation inconsistencies that must be corrected. These issues can be fixed with additional experiments and more detailed exposition, but the current form is not publishable without revision.

major comments (4)
  1. [§4.2, Tables 1 and 3] The central claim that GraphVideoAgent outperforms VideoAgent by 2.2 points on EgoSchema and 2.0 points on NExT-QA is not supported by any measure of statistical reliability. The system depends on stochastic GPT-4 generations and retrieval decisions, yet only single accuracy values are reported in Tables 1, 2, and 3. Under run-to-run variation, a 2-point gap on a benchmark like EgoSchema is often within the noise of the evaluation. Please report multiple seeds (at least 3–5) with standard deviations or confidence intervals, and ideally a paired significance test over videos for the main comparisons and for the ablations in Tables 5 and 6.
  2. [§3.2, Eq. (2) and §3.3, Eq. (3)] The method is under-specified in ways that prevent reproduction and independent evaluation. The global graph update function U in Eq. (2) is never defined; the text says new frames and relations are 'seamlessly integrated' but no algorithm is given for how nodes, edges, or state histories are added, merged, or pruned. Similarly, Eq. (3) defines the frame score with weights alpha=0.5, beta=0.3, gamma=0.2, but no sensitivity analysis or justification is provided, and the threshold tau=3 is stated without calibration. Please provide a precise algorithmic description of U (or pseudocode), the exact definitions of s_graph, s_visual, and s_temporal, and a sensitivity analysis for the weights and tau.
  3. [§3.3, confidence stopping rule] The adaptive retrieval loop stops when the LLM self-reports confidence level 3, but the paper gives no calibration or error analysis showing that level-3 self-reports actually correspond to correct answers. This rule is load-bearing for the claimed efficiency of using only ~8.2 frames: if the model is overconfident on partial evidence, early termination will systematically hurt accuracy. The baseline VideoAgent shares a similar rule, so this does not invalidate the graph's contribution, but it does affect the interpretation of the frame-efficiency result. Please report the distribution of confidence levels, the correlation between confidence and accuracy, and a breakdown of errors by stopping round.
  4. [§4.3, Table 6 and §4.1] The entity-scale ablation uses only 50 multiple-choice questions from a 'multi-entity hour-long video dataset' that is not described at all—its size, annotation process, video sources, and question categories are missing. With 50 questions, the reported differences (e.g., 64.0 vs. 60.0 for 2–3 entities, 52.0 vs. 42.0 for 7+ entities) have very large confidence intervals, and the claim of 'superior scalability' is not statistically supported. Please either describe this dataset fully and provide error bars or significance tests, or run the entity-scale analysis on a larger, existing benchmark with known properties.
minor comments (5)
  1. [Abstract] The abstract contains typos and missing spaces, e.g., 'GraphVideoAgentachieveda2.2%improvementoverexistingmethodswhilerequiringanalysisofonly8.2framesonaverage', and the improvements should be written as percentages (2.2% and 2.0%) for consistency with the rest of the paper.
  2. [§2.1] The citation 'VideoAgent [63]' is incorrect: reference [63] is 'A Simple LLM Framework for Long-Range Video Question-Answering' (LLoVi), not VideoAgent. This should be corrected, and the duplicate references [63] and [64] to the same paper should be merged.
  3. [Template/formatting] The manuscript contains placeholder ACM template text that should have been removed before submission: 'Conference acronym ’XX, June 03–05, 2018, Woodstock, NY', 'CCS Concepts Do Not Use This Code', and 'Make sure to enter the correct conference title from your rights confirmation email'. These artifacts indicate an incomplete submission and must be fixed.
  4. [§4.2] The sentence 'The performance difference between the subset (60.2%) and full test set (54.1%) reflects the distinct characteristics of our evaluation protocol' is confusing because those numbers appear to be VideoAgent's subset and full-set results, not GraphVideoAgent's. Rewrite this passage to clearly describe the two evaluation regimes and the reported numbers.
  5. [§4.1 and Table 6] The description of the multi-entity dataset in §4.1 is a single sentence with no details, and Table 6 does not state how the entity count per question was determined or how the 50 questions were selected. Please provide full dataset documentation and selection criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GraphVideoAgent is an empirical system whose gains are measured against external benchmarks, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's central claim is that an LLM agent with a dynamic entity-relation graph improves long-form video QA over sequential-memory baselines. This claim is supported by evaluations on EgoSchema and NExT-QA, whose ground-truth answers are external to the method and are never used to fit the system. The graph is constructed from uniformly sampled frame captions via named-entity recognition, dependency parsing, and LLM reasoning; the frame-scoring weights in Eq. (3) are stated as empirically set constants (alpha=0.5, beta=0.3, gamma=0.2) and are not fitted to benchmark labels. The adaptive stopping rule based on LLM self-reported confidence is a design choice shared with the baseline and is not a quantity derived from the target result. Citations to the authors' own prior work appear only in the related-work survey and are not load-bearing; the benchmark NExT-QA, co-authored by one of the present authors, serves only as an external evaluation set. No equation in the paper reduces by construction to a fitted value or to a self-citation. Therefore, no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method rests on the quality of pretrained models and on hand-set hyperparameters. No new physical entities are introduced. The free parameters are all engineering choices whose values are not systematically justified.

free parameters (5)
  • Frame scoring weights alpha, beta, gamma = alpha=0.5, beta=0.3, gamma=0.2
    Section 3.3, Eq (3). These weights are 'empirically set' to balance graph, visual, and temporal scores. No validation procedure is reported, so they act as free parameters chosen for the reported results.
  • Confidence threshold tau = 3 (level 3 sufficient)
    Section 3.3. The LLM agent stops retrieving frames once confidence reaches level 3. The threshold is hand-set and the reliability of the LLM's self-assessed confidence is not validated.
  • Temporal coherence weight alpha (Eq 1) = unspecified
    Section 3.2, Eq (1) defines T(e,f)=alpha*S(e,f)+(1-alpha)*R(e,f), but no value is given for alpha. This weight determines how state consistency and relation persistence are combined.
  • Initial uniform sample size N = unspecified
    Section 3.3. The system samples N frames to build the initial graph, but N is not reported. Only the final average frames per video, 8.2 and 8.1, is given, which is not enough to reconstruct the procedure.
  • Additional frames per retrieval stage = 3
    Section 3.3. Each retrieval stage adds at most 3 frames; this cap is a hand-set limit that affects efficiency and coverage.
assumptions (4)
  • domain assumption Pretrained vision-language and NLP models (EVA-CLIP, LaViLa, spaCy, GPT-4) produce sufficiently accurate captions, entity extractions, and reasoning for the graph to be meaningful.
    The entire graph is built from captions of sampled frames (Section 3.2). If the captioner misses or misdescribes entities, the graph will be incomplete or wrong.
  • domain assumption Dependency parsing of captions reliably identifies spatial, interaction, and action relations between entities.
    Section 3.2 states edges are constructed through dependency parsing of captions. This assumes linguistic parsing of auto-generated captions yields correct relations.
  • domain assumption The LLM's self-reported confidence on a three-level scale is a reliable indicator that the retrieved frames contain the evidence needed to answer the question.
    Section 3.3 uses confidence level 3 as the stopping criterion. No calibration or error analysis is provided for this self-assessment.
  • ad hoc to paper The graph update function U preserves the information needed for final answer generation and is stable across rounds.
    Section 3.2 defines G_{t+1}=U(G_t,F_{t+1},R_{t+1}) without specifying the update rule or properties. The mechanism is asserted rather than derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Long Videos via LLM-Powered Entity Relation Graphs." pith.science (2026). https://pith.science/paper/LC774AYP

@misc{pith2026250115953,
  author       = {Pith},
  title        = {Pith review of: Understanding Long Videos via LLM-Powered Entity Relation Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LC774AYP}},
  note         = {Machine review of arXiv:2501.15953}
}
read the original abstract

The analysis of extended video content poses unique challenges in artificial intelligence, particularly when dealing with the complexity of tracking and understanding visual elements across time. Current methodologies that process video frames sequentially struggle to maintain coherent tracking of objects, especially when these objects temporarily vanish and later reappear in the footage. A critical limitation of these approaches is their inability to effectively identify crucial moments in the video, largely due to their limited grasp of temporal relationships. To overcome these obstacles, we present GraphVideoAgent, a cutting-edge system that leverages the power of graph-based object tracking in conjunction with large language model capabilities. At its core, our framework employs a dynamic graph structure that maps and monitors the evolving relationships between visual entities throughout the video sequence. This innovative approach enables more nuanced understanding of how objects interact and transform over time, facilitating improved frame selection through comprehensive contextual awareness. Our approach demonstrates remarkable effectiveness when tested against industry benchmarks. In evaluations on the EgoSchema dataset, GraphVideoAgent achieved a 2.2 improvement over existing methods while requiring analysis of only 8.2 frames on average. Similarly, testing on the NExT-QA benchmark yielded a 2.0 performance increase with an average frame requirement of 8.1. These results underscore the efficiency of our graph-guided methodology in enhancing both accuracy and computational performance in long-form video understanding tasks.

Figures

Figures reproduced from arXiv: 2501.15953 by the authors.

Figure 1
Figure 1. Paradigm comparison — (a) The traditional method [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The figure illustrates GraphVideoAgent’s architecture, which consists of four main components: (1) an input module [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. GraphVideoAgent’s video analysis process has multiple components: a sequence of 8 video frames showing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The figure demonstrates a complete analysis pipeline for understanding why children leave a sofa. Starting with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The example demonstrates GraphVideoAgent’s improved frame selection efficiency compared to VideoAgent and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 48 canonical work pages

  1. [45]

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2025. Videoagent: Long-form video understanding with large language model as agent. InEuropean Conference on Computer Vision. Springer, 58–76

    work page 2025

  2. [63]

    InProceedings of the IEEE/CVF international conference on computer vision

    Just ask: Learning to answer questions from millions of narrated videos. InProceedings of the IEEE/CVF international conference on computer vision. 1686–1697

    work page

  3. [1]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    RohanAnil,SebastianBorgeaud,YonghuiWu,Jean-BaptisteAlayrac,Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Pau...

    arXiv doi:10.48550/arxiv.2312.11805 2023

  4. [2]

    Eysenck, and Michael C

    Alan Baddeley, Michael W. Eysenck, and Michael C. Anderson. 2020. Memory (3 ed.). Routledge. https://doi.org/10.4324/9780429449642

    work page doi:10.4324/9780429449642 2020

  5. [3]

    Ziyi Bai, Ruiping Wang, and Xilin Chen. 2024. Glance and Focus: Memory Prompting for Multi-Event Video Question Answering.Advances in Neural Information Processing Systems36 (2024)

    work page 2024

  6. [4]

    Ivana Balažević, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. 2024. Memory Consolidation En- ables Long-Context Video Understanding.arXiv preprint arXiv:2402.05861 (2024)

    arXiv 2024

  7. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.2020. Language models are few-shot learners. Advances in neural information processing systems33 (2020), 1877–1901

    work page 2020

  8. [6]

    Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022. Revisiting the" video" in video-language understanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2917–2927

    work page 2022

Show all 73 references
  1. [7]

    Meng Chu, Xuan Zhang, Zhedong Zheng, and Tat-Seng Chua. 2024. 3D- TAFS: A Training-free Framework for 3D Affordance Segmentation.arXiv preprint arXiv:2409.10078(2024)

    2024 arXiv

  2. [8]

    Meng Chu, Zhedong Zheng, Wei Ji, Tingyu Wang, and Tat-Seng Chua

  3. [9]

    Romain Cohendet, Claire-Hélène Demarty, Ngoc QK Duong, and Martin Engilberge. 2019. VideoMem: Constructing, analyzing, predicting short- term and long-term video memorability. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2531–2540

    2019

  4. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  5. [11]

    DifeiGao,LeiJi,LuoweiZhou,KevinQinghongLin,JoyaChen,ZihanFan, and Mike Zheng Shou. 2023. AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn.arXiv preprint arXiv:2306.08640 (2023)

    2023 arXiv

  6. [12]

    Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. 2023. MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14773–14783

    2023

  7. [13]

    Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. 2024. Graphdreamer: Compositional 3d scene synthesis from scene graphs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21295–21304

    2024

  8. [14]

    Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian

  9. [15]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14281– 14290

    2024

  10. [16]

    2020.spaCy: Industrial-strength Natural Language Processing in Python

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.spaCy: Industrial-strength Natural Language Processing in Python. https://spacy.io

    2020

  11. [17]

    VideoGraph:RecognizingMinutes-LongHumanActivitiesinVideos

    NoureldienHussein,EfstratiosGavves,andArnoldW.M.Smeulders.2019. VideoGraph:RecognizingMinutes-LongHumanActivitiesinVideos. CoRR abs/1905.05143 (2019). arXiv:1905.05143 http://arxiv.org/abs/1905.05143

    2019 arXiv

  12. [18]

    Md Mohaiminul Islam and Gedas Bertasius. 2022. Long Movie Clip Classi- fication with State-Space Video Models. InComputer Vision - ECCV 2022 - 17thEuropeanConference,TelAviv,Israel,October23-27,2022,Proceedings,Part XXXV (LectureNotesinComputerScience,Vol.13695) ,ShaiAvidan,Ga...

    2022 doi

  13. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Men- sch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guil- laume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-...

    2024 doi arXiv

  14. [20]

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2023. Chat-UniVi: Unified Visual Representation Empow- ers Large Language Models with Image and Video Understanding. arXiv:2311.08046 [cs.CV]

    2023 arXiv

  15. [21]

    Chenchen Jing, Yuwei Wu, Mingtao Pei, Yao Hu, Yunde Jia, and Qi Wu

  16. [22]

    Sungdong Kim, Jin-Hwa Kim, Jiyoung Lee, and Minjoon Seo. 2023. Semi- parametricvideo-groundedtextgeneration. arXivpreprintarXiv:2301.11507 (2023)

    2023 arXiv

  17. [23]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint arXiv:2401.13649(2024)

    2024 arXiv

  18. [24]

    Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zisserman, and Federico Tombari. 2023. Text-conditioned resampler for long form video understanding. arXiv preprint arXiv:2312.11897(2023)

    2023 arXiv

  19. [25]

    Berg, Mohit Bansal, and Jingjing Liu

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is More: ClipBERT for Video-and-Language Learningvia Sparse Sampling. InCVPR

    2021

  20. [26]

    KunChang Li,Yinan He,YiWang, YizhuoLi, Wenhai Wang, PingLuo,Yali Wang, Limin Wang, and Yu Qiao. 2023. VideoChat: Chat-Centric Video Understanding. arXiv:2305.06355 [cs.CV]

    2023 arXiv

  21. [27]

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. 2024. Visuala- gentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327(2024)

    2024 arXiv

  22. [28]

    Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. 2022. Ts2-net: Token shift and selection transformer for text-video retrieval. In European conference on computer vision. Springer, 319–335

    2022

  23. [29]

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang

  24. [30]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik

  25. [31]

    Karttikeya Mangalam, Ruslan Akshulakov, and Jitendra Malik. 2024. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances inNeural Information ProcessingSystems36 (2024)

    2024

  26. [32]

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. 2023. Verbs in action: Improving verb understanding in video-language models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 15579–15591

    2023

  27. [33]

    Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. 2022. S4nd: Modeling images and videos as multidimensional signals with state spaces.Advances in neural information processing systems35 (2022), 2846–2861

    2022

  28. [34]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] Understanding Long Videos via LLM-Powered Entity Relation Graphs Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

    2023 arXiv

  29. [35]

    EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tri...

    2023

  30. [36]

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisser- man, and Aida Nematzdeh. 2023. A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames. arXiv preprint arX...

    2023 arXiv

  31. [37]

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zis- serman, and Aida Nematzdeh. 2024. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames. InProceedings of t...

    2024

  32. [38]

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al

  33. [39]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389(2023)

    2023 arXiv

  34. [40]

    2021.Neuralscenegraphsfordynamicscenes.In ProceedingsoftheIEEE/CVF Conference on Computer Vision and Pattern Recognition

    Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. 2021.Neuralscenegraphsfordynamicscenes.In ProceedingsoftheIEEE/CVF Conference on Computer Vision and Pattern Recognition. 2856–2865

    2021

  35. [41]

    Long-formvideo-languagepre-trainingwithmultimodal temporal contrastive learning.Advances in neural information processing systems35 (2022), 38032–38045

    Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and JianlongFu.2022. Long-formvideo-languagepre-trainingwithmultimodal temporal contrastive learning.Advances in neural information processing systems35 (2022), 38032–38045

    2022

  36. [42]

    ViperGPT:VisualInfer- ence via Python Execution for Reasoning.Proceedings of IEEE International Conference on Computer Vision (ICCV)(2023)

    DídacSurís,SachitMenon,andCarlVondrick.2023. ViperGPT:VisualInfer- ence via Python Execution for Reasoning.Proceedings of IEEE International Conference on Computer Vision (ICCV)(2023)

    2023

  37. [43]

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. 2023. Selective Structured State-Spaces for Long- Form Video Understanding. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2...

    2023

  38. [44]

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. 2024. Vamos: Versatile Action Models for Video Under- standing. InComputer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XII (Lecture No...

    2024 doi

  39. [46]

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. 2024. EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters. CoRR abs/2402.04252 (2024). https://doi.org/10.48550/ ARXIV.2402.04252 arXiv:2402.04252

    2024 doi arXiv

  40. [47]

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022. InternVideo: General Video Foundation Models via Generative and Discriminativ...

    2022 doi arXiv

  41. [48]

    Yixin Wang, You Yang, and Mingzhuo Ren. 2023. Lifelongmemory: Lever- aging llms for answering queries in egocentric videos.arXiv preprint arXiv:2312.05269(2023)

    2023 arXiv

  42. [49]

    YuetianWeng,MingfeiHan,HaoyuHe,XiaojunChang,andBohanZhuang

  43. [50]

    Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. 2018. Linknet: Relational embedding for scene graph.Advances in neural informa- tion processing systems31 (2018)

    2018

  44. [51]

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2022. MeMViT: Memory- Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition.In IEEE/CVFConferenceonComputerVisionandPatternRecogni- tion...

    2022

  45. [52]

    SupervoxelAttentionGraphsforLong-Range Video Modeling

    Yang Wang, Gedas Bertasius, Tae-Hyun Oh, Abhinav Gupta, Minh Hoai, andLorenzoTorresani.2021. SupervoxelAttentionGraphsforLong-Range Video Modeling. InIEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021. IEEE, 155–166. http...

    2021

  46. [53]

    Contrastive VideoQuestionAnsweringviaVideoGraphTransformer

    J.Xiao,P.Zhou,A.Yao,Y.Li,R.Hong,S.Yan,andT.Chua.2023. Contrastive VideoQuestionAnsweringviaVideoGraphTransformer. IEEETransactions on Pattern Analysis; Machine Intelligence45, 11 (nov 2023), 13265–13280

    2023

  47. [54]

    Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. 2023. Retrieval-based Video Language Model for Efficient Long Video Question Answering. arXiv:2312.04931 [cs.CV]

    2023

  48. [55]

    AntoineYang,AntoineMiech,JosefSivic,IvanLaptev,andCordeliaSchmid

  49. [56]

    InEuropean Conference on Computer Vision

    Longvlm: Efficient long video understanding via large language models. InEuropean Conference on Computer Vision. Springer, 453–470

  50. [57]

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. 2022. Panoptic scene graph generation. InEuropean Conference on Computer Vision. Springer, 178–196

    2022

  51. [58]

    Jianing Yang, Yuying Zhu, Yongxin Wang, Ruitao Yi, Amir Zadeh, and Louis-Philippe Morency. 2020. What gives the answer away? question answeringbiasanalysisonvideoqadatasets. arXivpreprintarXiv:2007.03626 (2020)

    2020 arXiv

  52. [59]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next- qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9777–9786

    2021

  53. [60]

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2023. Self- Chained Image-Language Model for Video Localization and Question Answering. NeurIPS(2023)

    2023

  54. [61]

    Sangho Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2024. Self- chainedimage-languagemodelforvideolocalizationandquestionanswer- ing. Advances in Neural Information Processing Systems36 (2024)

    2024

  55. [62]

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. InProceedings of the IEEE conference on computer vision and pattern recognition. 5831–5840

    2018

  56. [64]

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022. Zero-Shot Video Question Answering via Frozen Bidi- rectional Language Models. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2...

    2022

  57. [65]

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. 2024. Flash-VStream: Memory-Based Real-Time Under- standing for Long Video Streams.arXiv preprint arXiv:2406.08085(2024)

    2024 arXiv

  58. [66]

    Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023. Learning video representations from large language models. InCVPR. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    2023

  59. [67]

    Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. 2023. Hitea: Hierarchical temporal-aware video-language pre- training. InProceedings of the IEEE/CVF International Conference on Computer Vision. 15405–15416

    2023

  60. [72]

    ChuntingZhang,ThomasLu,MdMohaiminulIslam,ZichenWang,Sangho Yu, Mohit Bansal, and Gedas Bertasius. 2023. A simple llm framework for long-range video question-answering.arXiv preprint arXiv:2312.17235 (2023)

    2023 arXiv

  61. [2020]

    InProceedings of the 28th ACM International Conference on Multimedia

    Visual-semantic graph matching for visual grounding. InProceedings of the 28th ACM International Conference on Multimedia. 4041–4050

  62. [2021]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Frameexit: Conditional early exiting for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15608–15618

  63. [2023]

    arXiv preprint arXiv:2312.08870(2023)

    Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens. arXiv preprint arXiv:2312.08870(2023)

    2023 arXiv

  64. [2024]

    18221–18232

    Moviechat: From dense token to sparse memory for long video understanding.In ProceedingsoftheIEEE/CVFConferenceonComputerVision and Pattern Recognition. 18221–18232

  65. [2025]

    InEuropean Conference on Computer Vision

    Towards natural language-guided drones: GeoText-1652 benchmark with spatial relation matching. InEuropean Conference on Computer Vision. Springer, 213–231

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.