pith. sign in

arxiv: 2607.00446 · v1 · pith:2SHBKNSLnew · submitted 2026-07-01 · 💻 cs.CV · cs.AI

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

Seohyun Lee , Seoung Choi , Dohwan Ko , Jongha Kim , Hyunwoo J. Kim This is my paper

Pith reviewed 2026-07-02 15:07 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords video corpus moment retrievalsoft query refinementiterative retrievaltemporal groundingagentic frameworklatent space refinementreinforcement learningmulti-turn reasoning
0
0 comments X

The pith

VideoSearch-R1 uses soft query refinement in continuous latent space to enable iterative video retrieval and temporal grounding when initial searches fail.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that video retrieval from large corpora should be treated as an iterative, refinable process rather than a one-shot preprocessing step, because failed initial retrievals otherwise doom downstream query-conditioned tasks like temporal grounding. It introduces Soft Query Refinement to adjust query tokens in continuous latent space instead of discrete text rewrites, allowing finer adjustments with fewer tokens, and trains the full loop end-to-end with Group Relative Policy Optimization using task-level rewards from retrieval and grounding performance. This matters for scaling video understanding as corpora grow, since agentic systems currently assume the right video is already provided and lack recovery mechanisms. The approach yields state-of-the-art results on three Video Corpus Moment Retrieval benchmarks by combining inter-video search refinement with intra-video reasoning in multi-turn interactions.

Core claim

VideoSearch-R1 is an agentic framework for iterative video retrieval and reasoning via multi-turn interaction with a video search engine. It introduces Soft Query Refinement (SQR) that refines search query tokens in continuous latent space rather than rewriting queries in discrete text space, and trains SQR and the reasoning process with Group Relative Policy Optimization guided by task-level reward signals from retrieval and downstream tasks. The method iteratively retrieves videos from large-scale corpora, refines search queries when needed, and performs precise query-conditioned temporal grounding within retrieved content, reaching state-of-the-art performance across three datasets on Vid

What carries the argument

Soft Query Refinement (SQR), which adjusts search query tokens directly in continuous latent space to enable efficient fine-grained corrections during iterative retrieval.

If this is right

  • Iterative retrieval with SQR recovers from initial failures that break one-shot pipelines.
  • SQR achieves comparable or better refinement than text rewriting while using significantly fewer generated tokens.
  • The trained multi-turn loop jointly optimizes retrieval and intra-video grounding end-to-end via task rewards.
  • Performance gains appear across three separate VCMR datasets.
  • The framework supports both inter-video search and intra-video temporal reasoning in the same agentic interaction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The latent refinement mechanism could transfer to other modalities such as image or audio corpus search where discrete query rewriting is costly.
  • Fewer generated tokens during refinement may translate to lower inference latency in production-scale video search systems.
  • Combining SQR with larger video foundation models might further tighten the grounding precision after each retrieval turn.

Load-bearing premise

That continuous latent-space adjustments to query tokens produce more effective retrieval corrections than either one-shot retrieval or discrete text rewriting when the initial search fails.

What would settle it

A controlled test on a VCMR dataset with deliberately low initial retrieval recall, measuring whether VideoSearch-R1's iterative SQR loop raises final grounding accuracy above strong non-iterative baselines that use the same underlying retriever and grounder.

Figures

Figures reproduced from arXiv: 2607.00446 by Dohwan Ko, Hyunwoo J. Kim, Jongha Kim, Seohyun Lee, Seoung Choi.

Figure 1
Figure 1. Figure 1: An illustrative example of VideoSearch-R1. As an agentic AI system, VideoSearch-R1 enables multi-turn interaction through iterative video retrieval and reasoning, leveraging an external video search engine. This pipeline unifies corpus-level inter-video reasoning (e.g., video retrieval) with intra-video reasoning (e.g., temporal grounding) grounded in the retrieved video. 15,19–22,41,43]. Although these ap… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison between hard query refinement and our Soft Query Re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Iterative video retrieval and reasoning of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of the number of soft tokens. R@1 is computed over samples with refined queries. 5 7 9 11 13 15 1 2 3 # Turns 0.3/R@1 0.5/R@1 0.7/R@1 IoU / R @ 1 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Changes in the retrieved video as the number of soft tokens increases. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 1
Figure 1. Figure 1: Illustration of three HQR methods. target video). To this end, we scale the query refiner from 2B to 8B in a simple retrieval-retry setting. When the initial top-1 retrieval is incorrect, the model rewrites the query and re-issues the retrieval. As shown in [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effect of query refinement methods on initially mismatched queries. Evaluated with Qwen3-VL￾Embedding-2B [3] and Qwen3-VL￾Embedding-8B [3] [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison between SQR and HQR. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
read the original abstract

As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes VideoSearch-R1, an agentic framework for iterative video retrieval from large-scale corpora and subsequent intra-video reasoning (e.g., query-conditioned temporal grounding). It introduces Soft Query Refinement (SQR) to adjust query tokens in continuous latent space (rather than discrete text rewriting), trained end-to-end via Group Relative Policy Optimization (GRPO) using task-level reward signals from retrieval and downstream tasks. The central claim is that this enables recovery from failed initial retrievals and yields state-of-the-art performance on Video Corpus Moment Retrieval (VCMR) across three datasets, with supporting analysis that SQR uses fewer tokens than text-based refinement. Code and checkpoints are released.

Significance. If the SOTA results and efficiency claims hold under rigorous evaluation, the work would be significant for video search and understanding by closing the gap between inter-video retrieval and intra-video reasoning in an iterative loop. The latent-space refinement and RL training with composite task rewards represent a coherent technical response to the stated limitation of one-shot retrieval. Public code availability is a clear strength that lowers verification risk.

major comments (2)
  1. [Abstract] Abstract: the central SOTA claim on VCMR across three datasets is stated without any quantitative results, baseline comparisons, ablation studies, or error analysis, rendering it impossible to assess whether the iterative loop delivers genuine gains or reduces to metric fitting.
  2. [Abstract] Abstract: the GRPO training description does not specify reward formulation details (e.g., how retrieval and grounding rewards are balanced or normalized), leaving open the circularity risk that reported improvements simply reflect direct optimization of the evaluation metrics rather than improved query refinement.
minor comments (2)
  1. [Abstract] Abstract: the three datasets are not named, preventing immediate assessment of task difficulty or comparison to prior VCMR literature.
  2. [Abstract] Abstract: the token-efficiency analysis is asserted but no metrics, comparison protocol, or example outputs are provided.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. We address each point below and will revise the abstract in the next version to improve clarity and support for the claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central SOTA claim on VCMR across three datasets is stated without any quantitative results, baseline comparisons, ablation studies, or error analysis, rendering it impossible to assess whether the iterative loop delivers genuine gains or reduces to metric fitting.

    Authors: We agree that the abstract, as currently written, lacks the quantitative anchors needed for independent assessment. The full manuscript reports specific mAP and Recall@K numbers on the three VCMR benchmarks (with comparisons to prior SOTA methods) in Table 1, along with ablations on the iterative loop and error analysis in Sections 4.3–4.5. In the revised version we will condense the key numerical results and a brief baseline comparison into the abstract itself so that the SOTA claim is no longer unsupported. revision: yes

  2. Referee: [Abstract] Abstract: the GRPO training description does not specify reward formulation details (e.g., how retrieval and grounding rewards are balanced or normalized), leaving open the circularity risk that reported improvements simply reflect direct optimization of the evaluation metrics rather than improved query refinement.

    Authors: The reward design (composite retrieval-plus-grounding reward, per-task normalization, and weighting coefficients) is fully specified in Section 3.3 and Algorithm 1. Nevertheless, the abstract’s brevity leaves the formulation opaque. We will add a concise clause to the abstract stating that GRPO is driven by task-level rewards derived from both retrieval and downstream grounding metrics (with full formulation in the main text). This should reduce the perceived circularity concern while remaining within abstract length limits. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper introduces SQR for latent-space query refinement and trains it end-to-end with GRPO using task-level rewards from retrieval and grounding; the reported SOTA on VCMR is presented as the empirical outcome of this loop rather than a quantity derived by construction from the inputs. No self-definitional equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no information on free parameters, axioms, or invented entities can be extracted.

pith-pipeline@v0.9.1-grok · 5843 in / 1059 out tokens · 35426 ms · 2026-07-02T15:07:23.485202+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 11 canonical work pages · 8 internal anchors

  1. [1]

    FirstName Alpher , title =

  2. [2]

    Journal of Foo , volume = 13, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe , title =. Journal of Foo , volume = 13, number = 1, pages =

  3. [3]

    Journal of Foo , volume = 14, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow , title =. Journal of Foo , volume = 14, number = 1, pages =

  4. [4]

    FirstName Alpher and FirstName Gamow , title =

  5. [5]

    Computer Vision -- ECCV 2022 , year =

    2022

  6. [6]

    arXiv preprint arXiv:2104.08860 , year=

    Clip4clip: An empirical study of clip for end to end video clip retrieval , author=. arXiv preprint arXiv:2104.08860 , year=

    work page arXiv

  7. [7]

    arXiv preprint arXiv:2209.06430 , year=

    Clip-vip: Adapting pre-trained image-text model to video-language representation alignment , author=. arXiv preprint arXiv:2209.06430 , year=

    work page arXiv

  8. [8]

    CVPR , year=

    X-pool: Cross-modal language-video attention for text-video retrieval , author=. CVPR , year=

  9. [9]

    ECCV , year=

    Ts2-net: Token shift and selection transformer for text-video retrieval , author=. ECCV , year=

  10. [10]

    ICML , year=

    Learning transferable visual models from natural language supervision , author=. ICML , year=

  11. [11]

    COLM , year=

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning , author=. COLM , year=

  12. [12]

    ICLR , year=

    MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS , author=. ICLR , year=

  13. [13]

    CVPR , year=

    Lamra: Large multimodal model as your advanced retrieval assistant , author=. CVPR , year=

  14. [14]

    EMNLP Findings , year=

    Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization , author=. EMNLP Findings , year=

  15. [15]

    ICCV , year=

    Bidirectional likelihood estimation with multi-modal large language models for text-video retrieval , author=. ICCV , year=

  16. [16]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    ECCV , year=

    Internvideo2: Scaling foundation models for multimodal video understanding , author=. ECCV , year=

  18. [18]

    NeurIPS , year=

    Think silently, think fast: Dynamic latent compression of llm reasoning chains , author=. NeurIPS , year=

  19. [19]

    NeurIPS , year=

    Latent Chain-of-Thought for Visual Reasoning , author=. NeurIPS , year=

  20. [20]

    ICML , year=

    Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning , author=. ICML , year=

  21. [21]

    ACL , year=

    Softcot: Soft chain-of-thought for efficient reasoning with llms , author=. ACL , year=

  22. [22]

    ICLR , year=

    Latent visual reasoning , author=. ICLR , year=

  23. [23]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    EMNLP , year=

    SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement , author=. EMNLP , year=

  25. [25]

    NeurIPS , year=

    Agile: A novel reinforcement learning framework of llm agents , author=. NeurIPS , year=

  26. [26]

    NeurIPS , year=

    Vrag-rl: Empower vision-perception-based rag for visually rich information understanding via iterative reasoning with reinforcement learning , author=. NeurIPS , year=

  27. [27]

    NeurIPS , year=

    Reagent-v: A reward-driven multi-agent framework for video understanding , author=. NeurIPS , year=

  28. [28]

    CVPR , year=

    Morevqa: Exploring modular reasoning models for video question answering , author=. CVPR , year=

  29. [29]

    EMNLP , year=

    Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents , author=. EMNLP , year=

  30. [30]

    CVPR , year=

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos , author=. CVPR , year=

  31. [31]

    ICLR , year=

    React: Synergizing reasoning and acting in language models , author=. ICLR , year=

  32. [32]

    NeurIPS , year=

    Verified: A video corpus moment retrieval benchmark for fine-grained video understanding , author=. NeurIPS , year=

  33. [33]

    NeurIPS , year=

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face , author=. NeurIPS , year=

  34. [35]

    NeurIPS , year=

    Video-r1: Reinforcing video reasoning in mllms , author=. NeurIPS , year=

  35. [36]

    OneThinker: All-in-one Reasoning Model for Image and Video

    Onethinker: All-in-one reasoning model for image and video , author=. arXiv preprint arXiv:2512.03043 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [37]

    CVPR , year=

    Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning , author=. CVPR , year=

  37. [39]

    COLM , year=

    Training large language models to reason in a continuous latent space , author=. COLM , year=

  38. [40]

    Representation Learning with Contrastive Predictive Coding

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [41]

    CVPR , year=

    Monet: Reasoning in latent visual space beyond images and language , author=. CVPR , year=

  40. [42]

    ECCV , year=

    Selective query-guided debiasing for video corpus moment retrieval , author=. ECCV , year=

  41. [43]

    ACM MM , year=

    Conquer: Contextual query-aware ranking for video corpus moment retrieval , author=. ACM MM , year=

  42. [44]

    CVPR , year=

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning , author=. CVPR , year=

  43. [45]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vtimellm: Empower llm to grasp video moments , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  44. [46]

    VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms , author=. arXiv preprint arXiv:2406.07476 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  45. [47]

    WACV , year=

    Flashvtg: Feature layering and adaptive score handling network for video temporal grounding , author=. WACV , year=

  46. [48]

    NeurIPS , year=

    Videochat-r1.5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception , author=. NeurIPS , year=

  47. [49]

    ICLR , year=

    Self-rag: Learning to retrieve, generate, and critique through self-reflection , author=. ICLR , year=

  48. [50]

    NeurIPS , year=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. NeurIPS , year=

  49. [51]

    NeurIPS , year=

    Video-rag: Visually-aligned retrieval-augmented long video comprehension , author=. NeurIPS , year=

  50. [52]

    CVPR , year=

    Cap4video: What can auxiliary captions do for text-video retrieval? , author=. CVPR , year=

  51. [53]

    Vide- orag: Retrieval-augmented generation with extreme long-context videos,

    Videorag: Retrieval-augmented generation with extreme long-context videos , author=. arXiv preprint arXiv:2502.01549 , year=

    work page arXiv

  52. [54]

    SIGIR , year=

    Video corpus moment retrieval with contrastive learning , author=. SIGIR , year=

  53. [55]

    NeurIPS , year=

    Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo , author=. NeurIPS , year=

  54. [56]

    AdaTooler-V: Adaptive Tool-Use for Images and Videos

    AdaTooler-V: Adaptive Tool-Use for Images and Videos , author=. arXiv preprint arXiv:2512.16918 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  55. [57]

    Qwen3-VL Technical Report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  56. [58]

    In: CVPR (2023)

    Li, J., Wei, P., Han, W., Fan, L.: Intentqa: Context-aware video intent reasoning. In: CVPR (2023)

    2023

  57. [59]

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

    Li, M., Zhang, Y., Long, D., Chen, K., Song, S., Bai, S., Yang, Z., Xie, P., Yang, A., Liu, D., et al.: Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720 (2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  58. [60]

    5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception , author=

    Videochat-r1. 5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception , author=. NeurIPS , year=

  59. [61]

    CVPR , year=

    Intentqa: Context-aware video intent reasoning , author=. CVPR , year=