Pith. sign in

REVIEW 3 major objections 126 references

Incentivizing Vision Language Models to Search for Long Video Question Answering

T0 review · 3 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Pith's one-line read Vision-language models answer long-video questions better when RL rewards them for retrieving a checklist of verifiable visual primitives, not just for the final answer.

desk verdict Agentic NL search + RL is the real lift; VETL is a sensible but modest dense-reward add-on whose causal role the abstract overstates. read the letter →

arxiv 2607.02959 v1 pith:EOIPWEHJ submitted 2026-07-03 cs.CV cs.LG

classification cs.CVcs.LG
keywords longvideoquestionansweringvision-languagemodelsagenticsearchreinforcementlearningtemporallogicneuro-symbolicrewardsvisualevidenceretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-video question answering usually fails because models see only a fixed, uniform sample of frames and therefore miss the sparse moments that actually matter. This paper turns the problem into multi-turn natural-language search: an agent issues search queries, receives short clips, reasons, and searches again until it answers. The hard part is teaching the agent what a good search trajectory looks like. Outcome-only rewards (final answer correct or not) are too sparse and invite shortcuts. The authors therefore compile each question into a temporal-logic specification that yields a finite checklist of atomic visual primitives—objects, actions, relations—then give the agent dense credit whenever those primitives appear in the clips it retrieved. Optimizing for this evidence-seeking behavior raises Pass@1 by up to 8% and Pass@4 by 15% over the same base model on long-video benchmarks, while using far fewer frames than uniform sampling or caption-heavy pipelines.

What carries the argument

VETL (Visual Evidence via Temporal Logic): an offline generate–judge loop that turns a question–answer pair into a temporal-logic formula over atomic visual primitives; the RL reward is then simply the fraction of those primitives that appear at least once in the agent’s retrieved clips (order ignored).

What would settle it

Train an otherwise identical agent with only exact-match answer reward on the same long-video suites; if Pass@1 and Pass@4 gains disappear (or reverse) on videos longer than ten minutes, the claim that the VETL evidence reward is what drives the improvement collapses.

Watch

Extended reading notes

Core claim

Post-training a vision-language agent with dense rewards derived from temporal-logic decompositions of the question (rather than answer accuracy alone) produces a policy that actively searches for the right visual evidence and therefore answers long-video questions more accurately and more sample-efficiently than outcome-only RL or passive frame sampling.

Load-bearing premise

That an LLM-written temporal-logic checklist plus imperfect visual detection of its atomic primitives is a faithful enough stand-in for “correct context” even though the reward discards ordering and the detector can be wrong.

Editorial extensions

If this is right

  • Agentic long-video systems can be improved by shaping intermediate search steps rather than waiting for final-answer supervision.
  • Natural-language search interfaces become learnable once a verifiable primitive checklist supplies dense feedback.
  • Frame budgets can drop well below the usual fixed 64-frame sampling while accuracy still rises.
  • The same neuro-symbolic reward recipe can be reused across GRPO, RLOO and related RL algorithms with only modest hyper-parameter retuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the primitive detector itself is weak, the reward becomes noisy; stronger open-vocabulary detectors or multi-modal verifiers would tighten the training signal without changing the overall architecture.
  • Discarding temporal order inside the reward may be fine for non-chronological search, but questions that truly hinge on sequence could benefit from a later stage that re-introduces order constraints.
  • The same checklist idea could be applied to other open-ended multimodal tools (audio, 3-D, long documents) wherever unit-test-style verification is currently missing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper presents VSeek, an agentic multi-turn framework that equips a VLM with natural-language search, subtitle search, and summary tools over a ViCLIP-indexed long video, then post-trains the policy with GRPO (and variants) so that the model jointly issues targeted queries and answers. To supply denser rewards than exact-match answer accuracy alone, the authors introduce VETL: an offline LLM+judge loop that compiles each question–answer pair into a temporal-logic formula over atomic visual primitives; the online reward (Eq. 6) is the fraction of those primitives detected by max-pooling ViCLIP scores over the retrieved clips (temporal operators are discarded). On LongVideoBench, Video-MME and MLVU the resulting VSeek-VETL model improves Pass@1 by up to ~9 % and Pass@4 by ~13 % relative to the Qwen3-VL-4B-Thinking base, with further gains on longer videos and a substantial reduction in frames processed.

Significance. If the claimed gains hold under stricter controls, the work supplies a practical recipe for turning long-video QA into an RL-search problem and demonstrates that a neuro-symbolic checklist can serve as a dense, offline-verifiable reward. The open-sourced code, multi-algorithm ablations (GRPO/RLOO/GSPO), frame-efficiency numbers, and qualitative traces are concrete assets for the community. The core engineering insight—that natural-language search plus a bag-of-primitives reward already yields large lifts—is useful even if the temporal-logic machinery contributes only marginally.

major comments (3)
  1. Abstract and §1 attribute the headline “up to 8 % Pass@1 / 15 % Pass@4” gains to “explicitly optimizing for this verifiable evidence-seeking behavior” via VETL. Table 1 shows that the bulk of the lift already appears in VSeek-EM (exact-match reward only): +8.8 % / +8.7 % Pass@1 on LongVideoBench and MLVU versus the same base model. VSeek-VETL adds only +0.4 / +1.2 / +2.1 points on the three ID sets and is mixed or negative on OOD. The causal contribution of the neuro-symbolic reward is therefore overstated; the paper should either rephrase the claim or supply a controlled ablation that isolates VETL’s incremental effect after agentic search is already present.
  2. §4.3 and Eq. (6) explicitly discard the temporal operators that Algorithm 1 and the abstract emphasize (“along with their temporal ordering”). The reward reduces to a bag-of-primitives max-detection score. Table 5 (VSeek-Direct) shows that even a non-TL decomposition yields nearly identical numbers, confirming that the formal temporal structure is not load-bearing. Either restore order-sensitive verification or remove the repeated claim that temporal logic is essential for the reward.
  3. No error bars, statistical tests, or multiple random seeds are reported for any table. Given that the VETL–EM deltas are often <2 points, it is impossible to judge whether they are reliable. At minimum the main ID comparisons (Table 1) and the RL-algorithm sweep (Table 4) need standard deviations or bootstrap intervals.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-level circularity: VETL rewards are computed offline from gold QA pairs via external LLM+judge and frozen ViCLIP detection; policy rollouts never observe gold answers or the reward model itself.

full rationale

The paper is an empirical methods contribution whose central results (Pass@k lifts on public LVQA benchmarks) are obtained by GRPO/RLOO/GSPO post-training of a VLM policy against a fixed, offline reward. Phase-1 TL specifications are generated once from (q, a_gold) by an external LLM+judge loop (Algorithm 1) and never updated by the policy; Phase-2 RVETL (Eq. 6) is a bag-of-primitives max-detection score under a frozen ViCLIP detector, deliberately discarding temporal order. The final scalar R = 0.7 Rem + 0.3 RVETL therefore supplies an independent training signal, exactly analogous to unit-test rewards in math/code RL. Empirical tables compare against the same base model under exact-match-only RL and against non-agentic baselines; no parameter is fitted on a subset and then re-presented as a prediction, no uniqueness theorem is imported from the authors to force the architecture, and no equation reduces to its own input by construction. Self-citations to the authors’ earlier neuro-symbolic video papers appear in Related Work and for the TL-generation prompt style, but they are not load-bearing for the measured accuracy numbers. The derivation chain is therefore self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central performance claim rests on standard RL (GRPO), off-the-shelf ViCLIP detection, LLM-generated TL specs, and a handful of hand-chosen mixing weights and thresholds. No new physical constants; the invented machinery is the VETL pipeline itself.

free parameters (3)
  • λ1, λ2 (reward mix) = 0.7 / 0.3
    Fixed at 0.7 / 0.3 for exact-match vs VETL; chosen by authors, not derived.
  • κ (judge acceptance threshold) = 5
    Score threshold for accepting a TL specification; set to 5.
  • max agent steps B, retriever top-k, clip length = 4 / 4 / 8s / 16
    Hard limits (B=4, top-k=4, 8 s clips, 16 frames) that shape the feasible policy.
assumptions (3)
  • domain assumption ViCLIP cosine similarity is a reliable binary detector Dp(ot) for the atomic visual primitives extracted from TL.
    Eq. 6 and Phase 2 treat the detector as ground truth for reward; no calibration or error model is supplied.
  • ad hoc to paper Temporal order of evidence retrieval can be ignored; only presence of primitives matters.
    Explicitly stated in §4.3 after introducing full TL operators; simplifies reward but discards part of the formal specification.
  • domain assumption GRPO (and RLOO/GSPO) with token masking of tool outputs yields stable policy improvement for multi-turn video search.
    Standard RL assumption imported from text-agent literature; Appendix shows GSPO actually degrades under VETL.
invented entities (2)
  • VETL (Visual Evidence via Temporal Logic) reward pipeline
    purpose: Compile NL questions into TL specs whose atomic primitives supply dense, verifiable intermediate rewards for RL.
    Core technical contribution; no independent external validation beyond the paper’s own ablations.
  • VSeek multi-tool agent loop
    purpose: Interleave <think>, natural-language <search>/<search_subtitle>/<summary>, and <answer> for long-video QA.
    Architectural wrapper around existing VLMs and ViCLIP index; novelty lies in the joint RL objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incentivizing Vision Language Models to Search for Long Video Question Answering." pith.science (2026). https://pith.science/paper/EOIPWEHJ

@misc{pith2026260702959,
  author       = {Pith},
  title        = {Pith review of: Incentivizing Vision Language Models to Search for Long Video Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOIPWEHJ}},
  note         = {Machine review of arXiv:2607.02959}
}
read the original abstract

We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.

Figures

Figures reproduced from arXiv: 2607.02959 by the authors.

Figure 1
Figure 1. Visualization of VSeek. Given an indexed long video and a question, VSeek iteratively decomposes the query, issues natural-language search requests to retrieve candidate clips, and refines its video context before producing the final answer. misses the sparse, query-critical moments [55,69,74] required to answer complex queries while utilizing irrelevant context that leads to incorrect answers [27]. To tackle this c… view at source ↗
Figure 2
Figure 2. VETL Rewards. We separately compile the question and its correct answer into a temporal logic specification in Phase 1. We utilize the extracted primitives from the temporal logic specification to verify whether the agent’s retrieved video context would satisfy answering the question in Phase 2. 3. Video Summary: Invoked via the <summary></summary> tag, this tool gives a global view of the video. When the agent lack… view at source ↗
Figure 3
Figure 3. Qualitative analysis. This example qualitatively highlights the advantage of VSeek-VETL over outcome-only based training. Here, both VSeek-EM and VSeek-VETL correctly decide that the first clip to retrieve must contain the desired subtitle. In contrast to VSeek-EM, which immediately hallucinates an answer, VSeek-VETL devises a more targeted search query to retrieve a set of segments corresponding to a pen holder to … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

126 extracted references · 3 canonical work pages

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  2. [2]

    In: Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers)

    Ahmadian, A., Cremer, C., Gall´ e, M., Fadaee, M., Kreutzer, J., Pietquin, O., ¨Ust¨ un, A., Hooker, S.: Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In: Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12248–12267 (2024)

  3. [3]

    arXiv preprint arXiv:1606.06565 (2016)

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Man´ e, D.: Con- crete problems in ai safety. arXiv preprint arXiv:1606.06565 (2016)

  4. [4]

    In: Proceedings of the IEEE International Con- ference on Computer Vision

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE International Con- ference on Computer Vision. pp. 2425–2433 (2015)

  5. [5]

    arXiv preprint arXiv:2511.21631 (2025) 16 H

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 16 H. Goel et al

  6. [6]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025),https://arxiv.org/abs/2502.13923

  7. [7]

    The MIT Press (2008)

    Baier, C., Katoen, J.P.: Principles of Model Checking. The MIT Press (2008)

  8. [8]

    Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? (2021),https://arxiv.org/abs/2102.05095

Show all 126 references
  1. [9]

    Advances in Neural Information Processing Systems37, 53168–53197 (2025)

    Chandrasegaran, K., Gupta, A., Hadzic, L.M., Kota, T., He, J., Eyzaguirre, C., Durante, Z., Li, M., Wu, J., Li, F.F.: Hourvideo: 1-hour video-language under- standing. Advances in Neural Information Processing Systems37, 53168–53197 (2025)

  2. [10]

    arXiv preprint arXiv:2412.12075 (2024)

    Chen, G., Liu, Y., Huang, Y., He, Y., Pei, B., Xu, J., Wang, Y., Lu, T., Wang, L.: Cg-bench: Clue-grounded question answering benchmark for long video under- standing. arXiv preprint arXiv:2412.12075 (2024)

  3. [11]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

  4. [12]

    arXiv preprint arXiv:2205.01089 (2022)

    Chen, Z., Yi, K., Li, Y., Ding, M., Torralba, A., Tenenbaum, J.B., Gan, C.: Com- phy: Compositional physical reasoning of objects and events from videos. arXiv preprint arXiv:2205.01089 (2022)

  5. [13]

    In: European Conference on Computer Vision

    Choi, M., Goel, H., Omama, M., Yang, Y., Shah, S., Chinchali, S.: Towards neuro- symbolic video understanding. In: European Conference on Computer Vision. pp. 220–236. Springer (2024)

  6. [14]

    arXiv preprint arXiv:2504.17180 (2025)

    Choi, M., Sharan, S., Goel, H., Shah, S., Chinchali, S.: We’ll fix it in post: Im- proving text-to-video generation with neuro-symbolic feedback. arXiv preprint arXiv:2504.17180 (2025)

  7. [15]

    arXiv preprint arXiv:2110.14168 (2021)

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

  8. [16]

    In: Handbook of Theoretical Com- puter Science, Volume B: Formal Models and Sematics (1991),https://api

    Emerson, E.A.: Temporal and modal logic. In: Handbook of Theoretical Com- puter Science, Volume B: Formal Models and Sematics (1991),https://api. semanticscholar.org/CorpusID:6062082

  9. [17]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recog- nition. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6202–6211 (2019)

  10. [18]

    In: IEEE Conference on Computer Vision and Pattern Recognition

    Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fu- sion for video action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 1933–1941. IEEE Computer Society, Las Vegas, NV, USA (2016)

  11. [19]

    arXiv preprint arXiv:2503.21776 (2025)

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025)

  12. [20]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern ...

  13. [21]

    arXiv preprint arXiv:2605.01248 (2026) VSeek 17

    Goel, H., Udathu, A., Jabireddy, S., Kalkar, P., Parulekar, A.: Sˆ 3-r1: Learn- ing to retrieve and answer step-by-step with synthetic data. arXiv preprint arXiv:2605.01248 (2026) VSeek 17

  14. [22]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

  15. [23]

    arXiv preprint arXiv:2401.14196 (2024)

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al.: Deepseek-coder: when the large language model meets programming– the rise of code intelligence. arXiv preprint arXiv:2401.14196 (2024)

  16. [24]

    In: 2019 IEEE 58th conference on decision and control (CDC)

    Hasanbeig, M., Kantaros, Y., Abate, A., Kroening, D., Pappas, G.J., Lee, I.: Re- inforcement learning for temporal logic control synthesis with probabilistic satis- faction guarantees. In: 2019 IEEE 58th conference on decision and control (CDC). pp. 5338–5343. IEEE (2019)

  17. [25]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, B., Li, H., Jang, Y.K., Jia, M., Cao, X., Shah, A., Shrivastava, A., Lim, S.N.: Ma-lmm: Memory-augmented large multimodal model for long-term video under- standing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13504–13514 (2024)

  18. [26]

    arXiv preprint arXiv:2509.24304 (2025)

    He, Z., Qu, X., Li, Y., Huang, S., Liu, D., Cheng, Y.: Framethinker: Learn- ing to think with long videos via multi-turn frame spotlighting. arXiv preprint arXiv:2509.24304 (2025)

  19. [27]

    URL https://www.trychroma.com/research/context-rot, retrieved October20, 2025 (2025)

    Hong, K., Troynikov, A., Huber, J.: Context rot: How increasing input tokens impacts llm performance. URL https://www.trychroma.com/research/context-rot, retrieved October20, 2025 (2025)

  20. [28]

    In: European conference on computer vision

    Huang, Q., Xiong, Y., Rao, A., Wang, J., Lin, D.: Movienet: A holistic dataset for movie understanding. In: European conference on computer vision. pp. 709–727. Springer (2020)

  21. [29]

    In: Proceed- ings of the Computer Vision and Pattern Recognition Conference

    Islam, M.M., Nagarajan, T., Wang, H., Bertasius, G., Torresani, L.: Bimba: Selective-scan compression for long-range video question answering. In: Proceed- ings of the Computer Vision and Pattern Recognition Conference. pp. 29096–29107 (2025)

  22. [30]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2758–2766 (2017)

  23. [31]

    arXiv preprint arXiv:2410.20252 (2024)

    Jeoung, S., Huybrechts, G., Ganesh, B., Galstyan, A., Bodapati, S.: Adaptive video understanding agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning. arXiv preprint arXiv:2410.20252 (2024)

  24. [32]

    Journal of Automated Rea- soning60, 43–62 (2018)

    Jha, S., Raman, V., Sadigh, D., Seshia, S.A.: Safe autonomy under perception uncertainty using chance-constrained temporal logic. Journal of Automated Rea- soning60, 43–62 (2018)

  25. [33]

    Jiang, F., Yuan, J., Tsaftaris, S.A., Katsaggelos, A.K.: Anomalous video event detection using spatiotemporal context. Comput. Vis. Image Underst.115(3), 323– 333 (2011).https://doi.org/10.1016/J.CVIU.2010.10.008,https://doi.org/ 10.1016/j.cviu.2010.10.008

  26. [34]

    arXiv preprint arXiv:2503.09516 (2025)

    Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., Han, J.: Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 (2025)

  27. [35]

    IEEE Transactions on Robotics25(6), 1370–1381 (2009).https://doi.org/10.1109/TRO.2009.2030225

    Kress-Gazit, H., Fainekos, G.E., Pappas, G.J.: Temporal-logic-based reactive mis- sion and motion planning. IEEE Transactions on Robotics25(6), 1370–1381 (2009).https://doi.org/10.1109/TRO.2009.2030225

  28. [36]

    In: Golenkov, V., Krasno- proshin, V., Golovko, V., Shunkevich, D

    Kroshchanka, A., Golovko, V., Mikhno, E., Kovalev, M., Zahariev, V., Zagorskij, A.: A neural-symbolic approach to computer vision. In: Golenkov, V., Krasno- proshin, V., Golovko, V., Shunkevich, D. (eds.) Open Semantic Technologies for Intelligent Systems. pp. 282–309. Springe...

  29. [37]

    Advances in neural information processing systems 33, 9459–9474 (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.t., Rockt¨ aschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, 9459–9474 (2020)

  30. [38]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

  31. [39]

    Neurocomputing490, 482–494 (2022)

    Li, N., Chang, F., Liu, C.: Human-related anomalous event detection via spatial- temporal graph convolutional autoencoder with embedded long short-term memory network. Neurocomputing490, 482–494 (2022)

  32. [40]

    arXiv preprint arXiv:2504.06958 (2025)

    Li, X., Yan, Z., Meng, D., Dong, L., Zeng, X., He, Y., Wang, Y., Qiao, Y., Wang, Y., Wang, L.: Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958 (2025)

  33. [41]

    arXiv preprint arXiv:2602.23553 (2026)

    Liang, S., Shah, S., Zhou, C., Sharan, S., Goel, H., Sanyal, A., Chinchali, S., Datta, G.: Le-neus: Latency-efficient neuro-symbolic video understanding via adaptive temporal verification. arXiv preprint arXiv:2602.23553 (2026)

  34. [42]

    In: European conference on computer vision

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024)

  35. [43]

    IEEE Transactions on Pattern Analysis and Machine Intelligence23(8), 873–889 (2001).https://doi.org/10.1109/34.946990

    Medioni, G., Cohen, I., Bremond, F., Hongeng, S., Nevatia, R.: Event detection and analysis from video streams. IEEE Transactions on Pattern Analysis and Machine Intelligence23(8), 873–889 (2001).https://doi.org/10.1109/34.946990

  36. [44]

    Automatica152, 110692 (2023).https://doi.org/https://doi.org/10.1016/j

    Mehdipour, N., Althoff, M., Tebbens, R.D., Belta, C.: Formal methods to comply with rules of the road in autonomous driving: State of the art and grand challenges. Automatica152, 110692 (2023).https://doi.org/https://doi.org/10.1016/j. automatica.2022.110692,https://www.scienc...

  37. [45]

    arXiv preprint arXiv:2511.18701 (2025)

    Munir, M., Goel, H., Wei, X., Choi, M., Shah, S., Bhardwaj, K., Whatmough, P., Chinchali, S., Marculescu, R.: Objectalign: Neuro-symbolic object consistency verification and correction. arXiv preprint arXiv:2511.18701 (2025)

  38. [46]

    arXiv preprint arXiv:2406.07400 (2024)

    Murphy, W., Holzer, N., Koenig, N., Cui, L., Rothkopf, R., Qiao, F., Santolucito, M.: Guiding llm temporal logic generation with explicit separation of data and control. arXiv preprint arXiv:2406.07400 (2024)

  39. [47]

    arXiv preprint arXiv:2511.05489 (2025)

    Pan, J., Zhang, Q., Zhang, R., Lu, M., Wan, X., Zhang, Y., Liu, C., She, Q.: Timesearch-r: Adaptive temporal search for long-form video understanding via self-verification reinforcement learning. arXiv preprint arXiv:2511.05489 (2025)

  40. [48]

    18th Annual Symposium on Founda- tions of Computer Science (FOCS) pp

    Pnueli, A.: The temporal logic of programs. 18th Annual Symposium on Founda- tions of Computer Science (FOCS) pp. 46–57 (1977)

  41. [49]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)

  42. [50]

    arXiv preprint arXiv:2308.12950 (2023)

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

  43. [51]

    In: CoCo@ NIPS (2015)

    Sarkar, S., Lore, K.G., Sarkar, S.: Early detection of combustion instability by neural-symbolic analysis on hi-speed video. In: CoCo@ NIPS (2015)

  44. [52]

    Advances in neural information processing systems36, 68539–68551 (2023)

    Schick, T., Dwivedi-Yu, J., Dess` ı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettle- moyer, L., Cancedda, N., Scialom, T.: Toolformer: Language models can teach VSeek 19 themselves to use tools. Advances in neural information processing systems36, 68539–68551 (2023)

  45. [53]

    arXiv preprint arXiv:1707.06347 (2017)

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  46. [54]

    arXiv preprint arXiv:2505.13851 (2025)

    Shah, S., Goel, H., Narasimhan, S.S., Choi, M., Sharan, S., Akcin, O., Chinchali, S.: A challenge to build neuro-symbolic video agents. arXiv preprint arXiv:2505.13851 (2025)

  47. [55]

    Proceedings of the AAAI Conference on Artifi- cial Intelligence40(11), 8805–8813 (Mar 2026).https://doi.org/10.1609/aaai

    Shah, S., Sharan, S.P., Goel, H., Choi, M., Munir, M., Pasula, M., Marculescu, R., Chinchali, S.: Neus-qa: Grounding long-form video understanding in temporal logic and neuro-symbolic reasoning. Proceedings of the AAAI Conference on Artifi- cial Intelligence40(11), 8805–8813 (...

  48. [56]

    arXiv preprint arXiv:2511.22570 (2025)

    Shao, Z., Luo, Y., Lu, C., Ren, Z., Hu, J., Ye, T., Gou, Z., Ma, S., Zhang, X.: Deepseekmath-v2: Towards self-verifiable mathematical reasoning. arXiv preprint arXiv:2511.22570 (2025)

  49. [57]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

  50. [58]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sharan, S., Choi, M., Shah, S., Goel, H., Omama, M., Chinchali, S.: Neuro-symbolic evaluation of text-to-video models using formal verification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8395– 8405 (2025)

  51. [59]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Song, E., Chai, W., Ye, T., Hwang, J.N., Li, X., Wang, G.: Moviechat+: Question- aware sparse memory for long video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

  52. [60]

    arXiv preprint arXiv:2503.05592 (2025)

    Song, H., Jiang, J., Min, Y., Chen, J., Chen, Z., Zhao, W.X., Fang, L., Wen, J.R.: R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592 (2025)

  53. [61]

    Tan, X., Luo, Y., Ye, Y., Liu, F., Cai, Z.: Allvb: All-in-one long video understanding benchmark (2025),https://arxiv.org/abs/2503.07298

  54. [62]

    In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition

    Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: Movieqa: Understanding stories in movies through question-answering. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 4631–4640 (2016)

  55. [63]

    arXiv preprint arXiv:2406.11409 (2024)

    Team, C., Zhao, H., Hui, J., Howland, J., Nguyen, N., Zuo, S., Hu, A., Choquette- Choo, C.A., Shen, J., Kelley, J., et al.: Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409 (2024)

  56. [64]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Tian, Y., Pang, G., Chen, Y., Singh, R., Verjans, J.W., Carneiro, G.: Weakly- supervised video anomaly detection with robust temporal feature magnitude learn- ing. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4975–4986 (2021)

  57. [65]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Tran, D., Wang, H., Torresani, L., Feiszli, M.: Video classification with channel- separated convolutional networks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5552–5561 (2019)

  58. [66]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, W., He, Z., Hong, W., Cheng, Y., Zhang, X., Qi, J., Ding, M., Gu, X., Huang, S., Xu, B., et al.: Lvbench: An extreme long video understanding benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22958–22967 (2025)

  59. [67]

    In: European Conference on Computer Vision

    Wang, X., Zhang, Y., Zohar, O., Yeung-Levy, S.: Videoagent: Long-form video understanding with large language model as agent. In: European Conference on Computer Vision. pp. 58–76. Springer (2024) 20 H. Goel et al

  60. [68]

    arXiv preprint arXiv:2307.06942 (2023)

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al.: Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 (2023)

  61. [69]

    In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., Bansal, M.: Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence. pp. 3272–3283 (2025)

  62. [70]

    Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

  63. [71]

    Advances in Neural Information Processing Systems38, 90925–90955 (2026)

    Xue, Z.S., Luo, R., Grauman, K.: Seeing the arrow of time in large multimodal models. Advances in Neural Information Processing Systems38, 90925–90955 (2026)

  64. [72]

    arXiv preprint arXiv:2309.10171 (2023)

    Yang, Y., Gaglione, J.R., Chinchali, S., Topcu, U.: Specification-driven video search via foundation models and formal verification. arXiv preprint arXiv:2309.10171 (2023)

  65. [73]

    In: The eleventh interna- tional conference on learning representations (2022)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: React: Synergizing reasoning and acting in language models. In: The eleventh interna- tional conference on learning representations (2022)

  66. [74]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J.C., Adeli, E., Fei-Fei, L., et al.: Re-thinking temporal search for long-form video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

  67. [75]

    In: Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval

    Ye, Y., Zhao, Z., Li, Y., Chen, L., Xiao, J., Zhuang, Y.: Video question answering via attribute-augmented attention network learning. In: Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. pp. 829–832 (2017)

  68. [76]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Yu, D., Yang, B., Wei, Q., Li, A., Pan, S.: A probabilistic graphical model based on neural-symbolic reasoning for visual relationship detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10599– 10608 (2022).https://doi.org/10.1109/CVP...

  69. [77]

    arXiv preprint arXiv:2506.10821 (2025)

    Yuan, H., Liu, Z., Zhou, J., Qian, H., Shu, Y., Sebe, N., Wen, J.R., Dou, Z.: Video- explorer: Think with videos for agentic long-video understanding. arXiv preprint arXiv:2506.10821 (2025)

  70. [78]

    arXiv preprint arXiv:2507.18071 (2025)

    Zheng, C., Liu, S., Li, M., Chen, X.H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., et al.: Group sequence policy optimization. arXiv preprint arXiv:2507.18071 (2025)

  71. [79]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al.: Mlvu: Benchmarking multi-task long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13691–13701 (2025)

  72. [80]

    arXiv preprint arXiv:2310.01852 (2023)

    Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., Li, Z., et al.: Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852 (2023)

  73. [81]

    Correct Answer

    Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L.: Visual7w: Grounded question an- swering in images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4995–5004 (2016) VSeek 21 A Appendix: Extra Results and Discussions A.1 Performance of Rewa...

  74. [82]

    ,→ ,→ ,→

    Reason: You must write your concise reasoning of 100-200 words within <think>...</think> that first summarizes and analyzes the frames to answer the question or to decide if you need to write a search query for more information. ,→ ,→ ,→

  75. [83]

    Decide: Based on your reasoning, decide if you have enough information in the frames obtained so far to either search for more information or answer the question. ,→ ,→

  76. [84]

    Act: At each turn you must choose ONE of the following actions: - If you need to search for more information using language based search to find objects or actions, output <search>...</search>,→ - If you need to search for frames that contain a specific subtitle, output <searc...

  77. [85]

    thinking

    Pouring flour 2. Adding ingredients 3. Pouring flour,→ <think>I should first locate where the chef uses a mixing bowl.</think>,→ <search>a chef with a large mixing bowl</search> tool (After the search tool call, the agent receives frames of the chef placing an empty bowl on th...

  78. [86]

    First think very concisely about the question and the provided images within the <think> and </think> tags.,→

  79. [87]

    You must provide the final answer the question with the correct option after ###.,→

  80. [88]

    Do not provide empty fields and you must provide an answer to the best of your ability.,→

  81. [89]

    <think>I have found the mixing bowl, but no ingredients have been added yet

    For example. <think>I have found the mixing bowl, but no ingredients have been added yet. I need to find the next action where something is put into the bowl. I found it, based on the question, the action is mixing, hence the answer option is 3.</think> ,→ ,→ ,→ ### 3 C Append...

  82. [90]

    A person in a blue swimsuit jumps into the water

  83. [91]

    A person in a black swimsuit jumps into the water

  84. [92]

    Two fully armed people walk on a forest path

  85. [93]

    The question is about what happens next after a man is surfing on a light blue surfboard with white waves

    A person in a swimsuit walks on the beach Continued on next page VSeek 31 T urn / Field Content Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. The question is about what happens next after a man is surfing on a light blue surfboard wit...

  86. [94]

    Alternatively, maybe the person is wearing a black swimsuit. So the correct answer would be 1.</think> <search>person in black swimsuit</search> Video Search T ool Response: Turn 3Reasoning/output:<think>Now that we’ve confirmed the person is in a black swimsuit, which matches...

  87. [95]

    The chef adds eggplant to the wok

  88. [96]

    The chef transfers the vegetables from the wok into a rectangular white plate containing bread

  89. [97]

    The chef adds scallions to the wok

  90. [98]

    The chef transfers the vegetables from the wok into a rectangular white plate containing rice

  91. [99]

    Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem

    The chef transfers the vegetables from the wok into a round white plate containing rice. Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. The question is about what happens after a chef pours a yellow liquid into a wok with stir-fried ve...

  92. [100]

    So the correct answer is either 3 or 4

    Let me check the options again. So the correct answer is either 3 or 4. Based on the possible frames, the correct one is 3 or 4. So the answer should be 3 or 4. Lets summarize. Reason: After pouring the yellow liquid, the next step is transferring the vegetables to a plate. Th...

  93. [101]

    He has both hands raised above his head

  94. [102]

    He has both hands crossed in front of his chest

  95. [103]

    He is doing the V-sign with both hands

  96. [104]

    So the question is about a man in a room with a bookshelf, wallpaper, and he’s wearing a red short-sleeve shirt sitting on a gaming chair

    He is doing the V-sign with one hand and clenching a fist with the other Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. So the question is about a man in a room with a bookshelf, wallpaper, and he’s wearing a red short-sleeve shirt sit...

  97. [105]

    The options are given as 0, 1, 2, 3, each with different sequences

    2→4→1→3 Original Video Continued on next page VSeek 35 T urn / Field Content Turn 1Reasoning/output:<think>Okay, let’s try to figure out the correct chrono- logical order of the events from the video. The options are given as 0, 1, 2, 3, each with different sequences. First, l...

  98. [106]

    So 4,2,1,3→option 2

    Alternatively, maybe the people are standing (4) first, then the man talks (2), then he’s on the lake (1), then kayaking (3). So 4,2,1,3→option 2. So which is correct? Let me think again. If the man is talking (2), then he’s on the lake side (1), which is part of the same scen...

  99. [107]

    cooking sausages→baking cookies→milking cow→making jewelry

  100. [108]

    making jewelry→cooking sausages→milking cow→baking cookies

  101. [109]

    cooking sausages→making jewelry→baking cookies→milking cow

  102. [110]

    The user is asking which option correctly matches the sequence of actions from the video

    baking cookies→making jewelry→milking cow→cooking sausages Original Video Continued on next page VSeek 37 T urn / Field Content Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. The user is asking which option correctly matches the sequence of actions f...

  103. [111]

    The black-haired woman in the white jacket

  104. [112]

    The black-haired woman in the olive-green jacket

  105. [113]

    The man in the black shirt

  106. [114]

    The woman in the olive-green trench coat

  107. [115]

    So, the question is asking which person in the scene has their head slightly bowed and is smiling

    The woman in the black dress Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. So, the question is asking which person in the scene has their head slightly bowed and is smiling. The options are 0 to 4. First, let’s break down the given in...

  108. [116]

    Continued on next page VSeek 39 T urn / Field Content Alternatively, the buyer (option 3) could be smiling

    is wearing black, but maybe she’s the one. Continued on next page VSeek 39 T urn / Field Content Alternatively, the buyer (option 3) could be smiling. But the description says the woman buying coffee is in the olive-green trench coat, which is option

  109. [117]

    Wait, but the user says the woman at the register is wearing black clothes, so that’s option

    Alternatively, maybe the woman in the black dress (option 4) is the one at the register, and she’s the one with the bowed head. Wait, but the user says the woman at the register is wearing black clothes, so that’s option

  110. [118]

    Let me check again

    Alternatively, perhaps the black-haired woman in the olive-green jacket (option 1) is the one. Let me check again. Alternatively, maybe the person is the one in the white jacket (option 0), but the description says the woman at the register is in black, and the buyer is in oli...

  111. [119]

    A doll wearing purple clothes

  112. [120]

    A yellow incense burner

  113. [121]

    A white chrysanthemum

  114. [122]

    The user is asking which object is present in the scene where there’s white text and a man in a black suit with a triangular shelf and a basketball

    A white dress Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. The user is asking which object is present in the scene where there’s white text and a man in a black suit with a triangular shelf and a basketball. First, the options are 0 ...

  115. [123]

    The user is asking for the type of movie the scene is in, with options 0 (Science Fiction), 1 (Comedy), 2 (Horror), 3 (Documentary)

    Documentary Original Video Turn 1Reasoning/output:<think>Okay, let’s try to figure out this problem. The user is asking for the type of movie the scene is in, with options 0 (Science Fiction), 1 (Comedy), 2 (Horror), 3 (Documentary). First, I need to determine which category t...

  116. [124]

    Besides the swimming pool, there is also a wooden dining table and some chairs, appears later

    Next to the swimming pool, there are white sunshades and trees, and there are some people inside the pool, appears first; A man wearing a blue short- sleeve shirt is taking pictures in front of the outdoor swimming pool. Besides the swimming pool, there is also a wooden dining...

  117. [125]

    Besides the swimming pool, there is also a wooden dining table and some chairs, appears later

    Next to the swimming pool, there are white sunshades and trees, and there are some people inside the pool, appears first; A man wearing a gray short- sleeve shirt is taking pictures in front of the outdoor swimming pool. Besides the swimming pool, there is also a wooden dining...

  118. [126]

    Besides the swimming pool, there is also a wooden dining table and some chairs, appears first

    A man wearing a gray short-sleeve shirt is taking pictures in front of the outdoor swimming pool. Besides the swimming pool, there is also a wooden dining table and some chairs, appears first. Next to the swimming pool, there are white sunshades and trees, and there are some p...

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.