Pith. sign in

REVIEW 4 major objections 3 minor 46 references

Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For ultra-long videos, a simple multi-key caption memory can beat pre-built graph memories on three QA benchmarks.

desk verdict Simple multi-key retrieval with inference-time expansion is a genuinely good idea and mostly backs its SOTA claims, but the Video-MME margin is thin and test-set-tuned hyperparameters blunt the headline. read the letter →

arxiv 2608.07663 v1 pith:NEBTPU4B submitted 2026-08-07 cs.CV cs.AIcs.CL

Yeeun Choi , Youngbeom Yoo , Joon-Young Lee , Hyolim Kang , Seon Joo Kim This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords ultra-longvideounderstandingquestionansweringretrieval-augmentedgenerationmulti-keymemoryepisodicneighborfilteringmulti-modallargelanguagemodelscaption-basedretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard recipe for ultra-long video QA—spend heavy computation before seeing the question to build hierarchical or graph memories—is backwards. Instead, it proposes storing each 30-second clip as a dense caption plus four complementary retrieval keys (event, dialogue, object, summary), and deferring all semantic composition to query time. At inference, matching the query against these keys picks anchor clips, and a neighbor-filtering step expands the temporal context around those anchors so the solver can reason over local evidence. The paper reports that this simpler pipeline sets new state-of-the-art accuracy on EgoLifeQA, LVBench, and Video-MME(Long), while using many fewer tokens than graph memory construction. The broad claim a sympathetic reader takes away: high-recall retrieval plus a strong reasoning model beats query-agnostic preprocessing.

What carries the argument

The load-bearing object is the episodic multi-key memory entry: a key-value record $(v_i, d_i, K_i)$ in which each clip carries exactly four lightweight textual keys. The matching mechanism is maximum-similarity retrieval over these keys, plus a symmetric temporal window $W_i = \{t : i - \Delta \le t \le i + \Delta\}$ around each retrieved clip, with the solver acting as a query-conditioned filter on the concatenated neighbor captions. This replaces pre-computed hierarchies or graphs with a flat store and on-demand local expansion.

What would settle it

Take a query whose answer depends on a detail absent from the caption of the correct 30-second clip—for example, the color of a small object visible only in frames—and force-retrieve exactly that clip; if the solver still cannot answer, the caption-completeness assumption fails and no amount of better retrieval can fix it.

Watch

Extended reading notes

Core claim

The central claim is that an episodic multi-key memory is sufficient for ultra-long video understanding, provided retrieval is high-recall and semantic composition happens at inference. Memory is built query-agnostically as $M = \{(v_i, d_i, K_i)\}$, where $v_i$ is a 30-second clip, $d_i$ its dense caption, and $K_i$ a set of four keys—event/action, dialogue/mention, object-state, and summary—that let different query intents match the same clip. Retrieval scores each clip by the maximum cosine similarity between the query and any of its keys, which boosts recall without hierarchical indexing. For each retrieved anchor, neighbor filtering concatenates captions from a temporal window of radius $\Delta = 2$ (roughly $\pm 1$ minute) and asks the solver to distill only query-relevant information. The paper reports state-of-the-art QA accuracy across three benchmarks with GPT-5 as solver—71.2% on EgoLifeQA, 71.8% on LVBench, and 77.7% on Video-MME(Long)—and argues that these results support deferring high-level relation composition to inference rather than pre-modeling it during memory construction.

Load-bearing premise

Everything downstream depends on the once-generated dense caption for each 30-second clip containing every visual and spoken detail a future question might ask about, because neither retrieval nor final reasoning can recover what the captioner omitted.

Editorial extensions

If this is right

  • Memory construction becomes a single pass: each clip is captioned once and keyed once, cutting input tokens by roughly 8.4 times and output tokens by 8.6 times relative to a multi-granularity graph memory baseline.
  • Fine-grained question types benefit most from retrieval quality: on EgoLifeQA, EntityLog and EventRecall improve sharply, while relation-heavy categories gain from temporal expansion.
  • Better solvers can absorb more retrieved clips: with GPT-5, raising TopN from 5 to 10 improves accuracy, whereas an open 8B solver degrades, so the optimal retrieval budget depends on solver capacity.
  • Most queries are answered early: 68.2% of EgoLifeQA questions finish in one retrieval round, averaging 1.72 rounds versus 2.45 for a graph-memory baseline, so simpler retrieval reduces inference steps as well as build cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the caption-omission assumption is the bottleneck, MERIT is a natural substrate for a hybrid that re-captions only the retrieved clips with a stronger vision model, rather than re-captioning the whole video per query.
  • The 'intelligence allocation' thesis generalizes beyond video RAG: any retrieval system with a strong reasoner may do better to preserve raw evidence and defer abstraction until the query is known, a claim that is testable in document and multimodal RAG settings.
  • The fixed radius $\Delta = 2$ is a place to adapt: since hit rate keeps rising to $\Delta = 4$ while accuracy plateaus, a query- or confidence-dependent temporal window could trade tokens for accuracy more efficiently.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes MERIT, a retrieval-based framework for ultra-long video QA. The method partitions a video into 30-second clips, generates a dense caption and four textual keys (event, dialogue, object, summary) per clip, retrieves top-N clips by maximum cosine similarity between the query embedding and the stored keys, expands each retrieved clip's temporal neighborhood by a radius Delta, applies a query-aware filter to the concatenated neighbor captions, and runs a multi-turn agentic loop before final QA on frames from the retrieved clips. The central claim is that this simple, query-agnostic memory construction with inference-time temporal expansion achieves state-of-the-art results on EgoLifeQA, LVBench, and Video-MME(Long), outperforming hierarchical and graph-based memory systems.

Significance. If the reported results hold, MERIT is a valuable counterpoint to the trend of increasingly complex hierarchical and graph memory systems: it shows that query-agnostic preprocessing can be kept lightweight while deferring semantic composition to inference time. The paper has clear strengths: the method is conceptually simple, the ablations in Tables 3, 4, 11, and 12 show that both multi-key retrieval and neighbor filtering contribute, and Table 6 documents substantial memory-construction efficiency gains over WorldMM. The EgoLifeQA (+5.6 points) and LVBench (+9.9 points) gains are large and, if confirmed, would be strong empirical support for the authors' hypothesis. However, the 'across three benchmarks' SOTA claim currently rests on a statistically fragile 1.1-point margin on Video-MME(Long), the headline hyperparameters are selected on the test set, and some reproducibility details (complete prompts, baseline re-evaluation, uncertainty quantification) are missing. These issues are fixable and do not invalidate the core idea, but they need to be addressed before the paper's strongest claims can be accepted.

major comments (4)
  1. [§4.3, Tables 2 and 5] The claim of state-of-the-art performance on Video-MME(Long) is not statistically supported. In Table 2, MERIT (GPT-5) scores 77.7 versus WorldMM (GPT-5) at 76.6 on 900 questions, a 1.1-point margin that could easily arise from sampling noise; no confidence intervals or paired significance tests (e.g., McNemar) are reported. The comparison is further weakened because the baseline numbers in Table 2 are taken from [22,33] rather than re-evaluated under the same protocol; indeed, the paper's own reproduction of WorldMM on EgoLifeQA (Table 5) gives 63.2, whereas Table 1 reports 65.6 for the same system. Please provide uncertainty quantification and a controlled re-evaluation of all baselines on all three benchmarks, or temper the 'across three benchmarks' claim in the abstract.
  2. [Tables 7 and 12] The default settings TopN=10 and Delta=2 are selected by sweeping on the EgoLifeQA test set (Table 7 sweeps TopN; Table 12 sweeps the neighbor-filtering radius), and the same settings are then used for the headline results in Tables 1 and 2. This test-set tuning introduces optimism that is not accounted for in the reported +5.6 and +9.9 gains. The paper should either use a held-out validation set, a nested selection protocol, or a sensitivity analysis showing that the qualitative conclusions are stable across a range of TopN and Delta values.
  3. [§3.2, Eq. (3), and Supp. E.2] The memory representation in Eq. (3) stores, for each 30-second clip, a dense caption d_i and derived keys K_i; retrieval and final QA can only access visual frames from the few retrieved clips. Consequently, any information omitted by the GPT-5-mini captions—small object attributes, subtle actions, brief spoken exchanges—is unrecoverable. This caption-coverage ceiling is a fundamental scope limitation and should be stated in the main text; the failure case in Supp. E.2 (frequency/counting queries) is one manifestation and is currently only discussed in the supplement.
  4. [Supp. A and F] The paper states in Section A that 'all specific LLM prompts used in this pipeline are provided in Section F,' but Section F contains only the multi-key extraction and neighbor-filtering prompts. The prompts for retrieval-query formulation, evidence-sufficiency assessment, and final QA generation—central to the agentic loop—are missing, and the manuscript does not describe a code/data release. Please make the complete prompt set and decoding settings available so that the reported numbers can be reproduced.
minor comments (3)
  1. [Fig. 2] Figure 2 contains garbled labels (e.g., 'ultra-ong', 'ltra-ong', 'Captionin', 'Similarit') that should be corrected.
  2. [References] Reference [10] lists arXiv:2410.057792(3), which appears to contain a typo in the arXiv identifier.
  3. [§4.3] The sentence 'the gap expands from +3.4% on Video-MME(L) to an impressive +11.4% on LVBench' compares MERIT to GPT-5 with uniform sampling, not to the previous memory-based SOTA; the wording should distinguish these comparisons to avoid implying a 3.4-point gain over the prior SOTA.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MERIT's headline results are benchmark measurements produced by an independently specified retrieval pipeline; the only notable issue is test-set hyperparameter selection, which is a validity concern rather than circularity.

full rationale

The paper contains no derivation chain whose output reduces to its inputs. The memory representation M = {(v_i, d_i, K_i)} (Eq. 3), the four key types (Eq. 5), the maximum-similarity retrieval score (Eqs. 6-7), and the neighbor-filtering window (Eqs. 8-9) are constructive definitions of an algorithm; none of them logically entails the accuracy values 71.2 on EgoLifeQA, 71.8 on LVBench, or 77.7 on Video-MME(Long). Those numbers are obtained by running the pipeline on externally labeled benchmarks and comparing against baselines from prior work (Tables 1-2), not by solving for a quantity that was assumed in the setup. The method does not fit its retrieval or reasoning parameters to the test answers; the only tunable choices are TopN and the neighbor window Delta, selected from Tables 7 and 12 on the EgoLifeQA test split. This is test-set tuning and could inflate the reported EgoLifeQA margin, and reusing those settings on the other benchmarks introduces mild selection optimism, but it is not circular: the final accuracy is a measurement, not a quantity forced by construction from the selected hyperparameters. The Video-MME(Long) margin over WorldMM (77.7 vs. 76.6) is small and reported without error bars, which is an uncertainty/statistical-support concern, not a circularity concern. No load-bearing self-citation or imported uniqueness theorem appears; the framework is self-contained and evaluated against external systems. Therefore no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are operational choices (clip length, TopN, retrieval rounds, neighbor radius, frame sampling) that are set on the evaluation benchmarks, with the most consequential being TopN and Delta. The key epistemic load falls on caption completeness and embedding alignment, both inherited from the caption-based RAG paradigm.

free parameters (5)
  • Neighbor filtering radius Delta = 2 (default; Delta=4 yields 72.2 avg on EgoLifeQA with GPT-5)
    Chosen from ablation on the evaluation benchmark (Table 12). Larger radii increase hit rate and accuracy but inflate inference tokens; the default is a test-set-tuned efficiency/accuracy tradeoff.
  • TopN retrieved clips = 10 for GPT-5 and Gemini 2.5 Pro; 5 for Qwen3-VL-4B/8B
    Selected per solver based on EgoLifeQA accuracy (Table 7). Proprietary solvers improve with 10 clips, while open models degrade with 10.
  • Maximum retrieval rounds = 5
    Agentic loop cap; Figure 4 shows accuracy converges by 5 rounds. This is a test-set choice, though mild and standard.
  • Clip length = 30 seconds
    Fixed segmentation granularity inherited from the captioning setup. It is load-bearing because all retrieval, keys, and neighbor expansion operate on this unit, but it is not fitted per dataset.
  • Frame sampling schedule for final QA = Round 1: 6 frames/clip for top 10 clips; Round 2: 6 frames for top 5; Round 3+: 1 frame/clip
    Adaptive visual sampling schedule described in Appendix A; chosen for cost and not ablated in the main paper.
assumptions (4)
  • domain assumption GPT-5-mini generated dense captions for each 30-second clip contain every piece of visual and dialogue evidence needed to answer benchmark queries.
    Section 3.2: memory is built only from captions; retrieval queries match keys derived from captions, and the final answer uses frames only from retrieved clips. Caption omissions are unrecoverable.
  • domain assumption Cosine similarity between Qwen3-Embedding-4B embeddings of the query and of extracted keys is a sufficient semantic match for retrieval.
    Appendix A: retrieval uses Qwen3-Embedding-4B; no calibration or query-type-specific validation is reported beyond final QA accuracy and hit rate.
  • domain assumption Evidence relevant to a query is temporally local, contained within the anchor clip plus a fixed radius of adjacent clips.
    Section 3.4 and Table 12: neighbor filtering assumes a window of +/-2 clips is enough; hit rate and accuracy vary with the window, so this is a tuned assumption rather than an independently established property.
  • domain assumption Benchmark annotations, target timestamps, and multiple-choice labels are correct, and accuracy on them reflects video understanding.
    Sections 4.1 and 4.3 use EgoLifeQA, LVBench, and Video-MME as ground truth; no audit of annotation quality is performed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding." pith.science (2026). https://pith.science/paper/NEBTPU4B

@misc{pith2026260807663,
  author       = {Pith},
  title        = {Pith review of: Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NEBTPU4B}},
  note         = {Machine review of arXiv:2608.07663}
}
read the original abstract

When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs). This ultra-long setting necessitates a two-stage paradigm: query-agnostic memory construction followed by retrieval-based inference. Prior work invests in complex memory construction to pre-model high-level relations in videos, despite not knowing the downstream query at build time. We instead prioritize high-recall retrievability during memory building, and defer query-specific, high-level relation composition to inference time. To this end, we propose MERIT(Multi-key Episodic Retrieval with Inference-time Temporal expansion), a simple yet effective agentic framework for ultra-long video understanding. First, we formulate an episodic multi-key representation that enables precise retrieval of fine-grained memories through a simple key-matching mechanism. Second, we introduce a neighbor filtering mechanism to capture broader semantic context without the massive computational overhead of global memory construction. This is achieved by expanding the temporal scope exclusively around the retrieved segments at inference time. By leveraging simple key-matching with this on-demand temporal expansion, MERIT achieves state-of-the-art performance across three long-video benchmarks: EgoLifeQA, LVBench, and Video-MME (Long).

Figures

Figures reproduced from arXiv: 2608.07663 by the authors.

Figure 1
Figure 1. (a) Previous methods: Construct hierarchical graph-based episodic and seman￾tic memories, and at inference time, search and aggregate relevant information across these structured memories. (b) Our method: MERIT constructs a simple episodic multi￾key memory, and at inference time, retrieves the matched clips along with temporal expansion to perform query-driven semantic reasoning. (c) Performance of MERIT against Ego… view at source ↗
Figure 2
Figure 2. Overall Pipeline of MERIT. (a) Stage 1: Episodic memory construction. For each 30-second video clip, dense captions are generated. Subsequently, an episodic multi-key extraction process derives four distinct keys per clip, collectively forming the memory M. (b) Stage 2: Agentic retrieval and answering. The solver regenerates a query and matches it against multi-keys to retrieve the most relevant clips. Neighbor Filt… view at source ↗
Figure 3
Figure 3. Qualitative results of MERIT on EgoLifeQA [31]. (a) The event key precisely matches the query, enabling retrieval of the exact target segment. (b) While both set￾tings retrieve the correct segment, neighbor filter supplies the surrounding ±∆ context necessary to resolve relational queries that cannot be answered from a single clip alone. individuals. Neighbor filtering resolves this by expanding the temporal context… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Retrieval round distribution and performance scaling in MERIT. (a) The dis￾tribution of required retrieval rounds to answer a query across different benchmarks. (b) Average accuracy scaling against the maximum round budget. C.4 Analysis of Retrieval Rounds We present a…
Figure 5
Figure 5. Figure 5: Qualitative comparison on a RelationMap query, where MERIT successfully resolves complex interpersonal interactions using Neighbor Filtering, while other base￾lines fail [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results for a query with an extreme temporal gap, highlighting that MERIT efficiently retrieves the correct context in a single round via the Event Key [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]
Figure 9
Figure 9. Figure 9: Prompt used for Multi-key Extraction [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]
Figure 10
Figure 10. Figure 10: Prompt used for Neighbor Filtering [PITH_FULL_IMAGE:figures/full_fig_p033_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 22 canonical work pages

  1. [1]

    In: Proceedings of the 33rd ACM Inter- national Conference on Information and Knowledge Management

    Arefeen, M.A., Debnath, B., Uddin, M.Y.S., Chakradhar, S.: irag: Advancing rag for videos with an incremental approach. In: Proceedings of the 33rd ACM Inter- national Conference on Information and Knowledge Management. pp. 4341–4348 (2024)

    work page 2024

  2. [2]

    In: European Conference on Computer Vision

    Ataallah, K., Shen, X., Abdelrahman, E., Sleiman, E., Zhuge, M., Ding, J., Zhu, D., Schmidhuber, J., Elhoseiny, M.: Goldfish: Vision-language understanding of arbitrarily long videos. In: European Conference on Computer Vision. pp. 251–

    work page

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    arXiv 2025

  4. [4]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  5. [5]

    arXiv preprint arXiv:2412.05271 (2024)

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    arXiv 2025

  7. [7]

    arXiv preprint arXiv:2404.16130 (2024)

    Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R.O., Larson, J.: From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024)

    arXiv 2024

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24108–24118 (2025)

    2025

Show all 46 references
  1. [9]

    arXiv preprint arXiv:2312.10997 (2023)

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    2023 arXiv

  2. [10]

    arXiv preprint arXiv:2410.057792(3) (2024)

    Guo, Z., Xia, L., Yu, Y., Ao, T., Huang, C.: Lightrag: Simple and fast retrieval- augmented generation. arXiv preprint arXiv:2410.057792(3) (2024)

    2024 arXiv

  3. [11]

    arXiv preprint arXiv:2502.14802 (2025)

    Gutiérrez, B.J., Shu, Y., Qi, W., Zhou, S., Su, Y.: From rag to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802 (2025)

    2025 arXiv

  4. [12]

    In: Findings of the Association for Computational Linguis- tics: ACL 2025

    Jeong, S., Kim, K., Baek, J., Hwang, S.J.: Videorag: Retrieval-augmented genera- tion over video corpus. In: Findings of the Association for Computational Linguis- tics: ACL 2025. pp. 21278–21298 (2025)

    2025

  5. [13]

    In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval

    Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via con- textualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 39–48 (2020)

    2020

  6. [14]

    Advances in neural information processing systems 33, 9459–9474 (2020) MERIT 17

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, 9459–9474 (2020) MERIT 17

    2020

  7. [15]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:2504.10739 (2025)

    Lin, Y., Wang, Q., Ye, H., Fu, Y., Li, H., Chen, Y., et al.: Hippomm: Hippocampal- inspired multimodal memory for long audiovisual event understanding. arXiv preprint arXiv:2504.10739 (2025)

    2025

  9. [17]

    Long, L., He, Y., Ye, W., Pan, Y., Lin, Y., Li, H., Zhao, J., Li, W.: Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory (2025)

    2025

  10. [18]

    Luo, Y., Zheng, X., Li, G., Yin, S., Lin, H., Fu, C., Huang, J., Ji, J., Chao, F., Luo, J., et al.: Video-rag: Visually-aligned retrieval-augmented long video compre- hension (2024)

    2024

  11. [19]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ma,Z.,Gou,C.,Shi,H.,Sun,B.,Li,S.,Rezatofighi,H.,Cai,J.:Drvideo:Document retrieval based long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18936–18946 (2025)

    2025

  12. [20]

    OpenAI: Gpt-5 system card (August 2025),https://openai.com/index/gpt-5- system-card/

    2025

  13. [21]

    In: CVPR

    Reddy, A., Martin, A., Yang, E., Yates, A., Sanders, K., Murray, K., Kriz, R., de Melo, C.M., Van Durme, B., Chellappa, R.: Video-colbert: Contextualized late interaction for text-to-video retrieval. In: CVPR. pp. 19691–19701 (2025)

    2025

  14. [22]

    Rege, A., Sadhu, A., Li, Y., Li, K., Vinayak, R.K., Chai, Y., Lee, Y.J., Kim, H.J.: Agentic very long video understanding (2026)

    2026

  15. [23]

    arXiv preprint arXiv:2502.01549 (2025)

    Ren, X., Xu, L., Xia, L., Wang, S., Yin, D., Huang, C.: Videorag: Retrieval-augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549 (2025)

    2025 arXiv

  16. [24]

    arXiv preprint arXiv:2410.17434 (2024)

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., Liu, Z., Xu, H., Kim, H.J., Soran, B., Krishnamoorthi, R., Elhoseiny, M., Chandra, V.: Longvu: Spatiotemporal adaptive compression for long video-language understanding. a...

    2024 arXiv

  17. [25]

    Shen, X., Zhang, W., Chen, J., Elhoseiny, M.: Vgent: Graph-based retrieval- reasoning-augmented generation for long video understanding (2025)

    2025

  18. [26]

    arXiv preprint arXiv:2506.06144 (2025)

    Wan, D., Wang, H., Stengel-Eskin, E., Cho, J., Bansal, M.: Clamr: Contextualized late-interaction for multimodal content retrieval. arXiv preprint arXiv:2506.06144 (2025)

    2025 arXiv

  19. [27]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang,W.,He,Z.,Hong,W.,Cheng,Y.,Zhang,X.,Qi,J.,Ding,M.,Gu,X.,Huang, S., Xu, B., et al.: Lvbench: An extreme long video understanding benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22958–22967 (2025)

    2025

  20. [28]

    In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., Bansal, M.: Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence. pp. 3272–3283 (2025)

    2025

  21. [29]

    Xu, Z., Zhang, J., Wang, Q., Liu, Y.: E-vrag: Enhancing long video understanding with resource-efficient retrieval augmented generation (2025)

    2025

  22. [30]

    arXiv preprint arXiv:2506.13589 (2025)

    Xue, Z., Zhang, J., Xie, X., Cai, Y., Liu, Y., Li, X., Tao, D.: Adavideorag: Omni- contextual adaptive retrieval-augmented efficient long video understanding. arXiv preprint arXiv:2506.13589 (2025)

    2025

  23. [31]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yang, J., Liu, S., Guo, H., Dong, Y., Zhang, X., Zhang, S., Wang, P., Zhou, Z., Xie, B., Wang, Z., et al.: Egolife: Towards egocentric life assistant. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28885–28900 (2025)

    2025

  24. [32]

    In: Proceedings of the Computer Vision and Pattern Recog- nition Conference (2025) 18 Y

    Yang, J., et al.: Ego-r1. In: Proceedings of the Computer Vision and Pattern Recog- nition Conference (2025) 18 Y. Choi et al

    2025

  25. [33]

    arXiv preprint arXiv:2512.02425 (2025)

    Yeo, W., Kim, K., Yoon, J., Hwang, S.J.: Worldmm: Dynamic multimodal memory agent for long video reasoning. arXiv preprint arXiv:2512.02425 (2025)

    2025

  26. [34]

    arXiv preprint arXiv:2501.13106 (2025)

    Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., et al.: Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 (2025)

    2025 arXiv

  27. [35]

    arXiv preprint arXiv:2505.24869 (2025)

    Zhang, C., Lin, Y.B., Wang, Z., Bansal, M., Bertasius, G.: Silvr: A simple language- based video reasoning framework. arXiv preprint arXiv:2505.24869 (2025)

    2025 arXiv

  28. [36]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Zhang, C., Lu, T., Islam, M.M., Wang, Z., Yu, S., Bansal, M., Bertasius, G.: A simple llm framework for long-range video question-answering. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 21715–21737 (2024)

    2024

  29. [37]

    arXiv preprint arXiv:2506.05176 (2025)

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al.: Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025)

    2025 arXiv

  30. [38]

    Zhang, Y., Li, B., Liu, h., Lee, Y.j., Gui, L., Fu, D., Feng, J., Liu, Z., Li, C.: Llava-next: A strong zero-shot video understanding model (April 2024),https: //llava-vl.github.io/blog/2024-04-30-llava-next-video/

    2024

  31. [39]

    time ref- erence

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024) MERIT 19 Supplementary Materials This supplementary material provides additional details and analyses that com- plement...

    2024 arXiv

  32. [40]

    - Use one short sentence or phrase

    Event / Action key - What physical actions or events actually happened between people? - Focus on observable actions and interactions. - Use one short sentence or phrase. - Include the agent if identifiable (use actual names if present). - If the speaker uses first-person expr...

  33. [41]

    - Use one short sentence or phrase

    Dialogue / Mention key - What was said, asked, or mentioned in the dialogue? - Focus on questions, statements, commands, or repeated mentions. - Use one short sentence or phrase

  34. [42]

    - Use one short sentence or phrase

    Object-state / Item-centric key - What object was handled, requested, moved, or referenced? - Describe the object and its state or role in the scene. - Use one short sentence or phrase

  35. [43]

    - Abstract away redundant or repeated actions

    Summary / Retrieval key - Generate ONE concise retrieval key that best represents the core event of the clip. - Abstract away redundant or repeated actions. - Capture the main entities, actions, and intent. - Be concise and retrieval-friendly. - Prefer compact keyword-style ph...

  36. [44]

    For each retrieved clip, analyze its 5 neighbor captions (before_2, before_1, center, after_1, after_2)

  37. [45]

    Extract ONLY the information relevant to answering the question

  38. [46]

    clip_1":

    Return relevant info for each clip Output format (JSON): { "clip_1": "...concise relevant info...", "clip_2": "", ... "clip_N": "..." } Guidelines: - Focus on information that directly helps answer the question - If no relevant information is found for a clip, output empty str...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.