Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval

Cheol-Ho Cho; Jae-Pil Heo; Jihyun Lee; Woojin Jeong; Woojin Jun

REVIEW 2 major objections 6 minor 52 references

Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zero-shot video moment retrieval works better when candidate moments are proposed from a video's own internal self-similarity, not from query-to-frame matching, and then verified by a multimodal language model.

desk verdict Useful training-free ZMR method with honest ablations, but the SOTA claim overreaches outside QVHighlights and the query-agnostic proposal stage needs an oracle-recall check. read the letter →

arxiv 2607.19027 v1 pith:2URLOLOP submitted 2026-07-21 cs.CV

Jihyun Lee , Cheol-Ho Cho , Woojin Jun , Woojin Jeong , Jae-Pil Heo This is my paper · ORCID

classification cs.CV

keywords zero-shotvideomomentretrievalself-similaritytemporalboundarydetectionmodalitygaplanguage-styleMLLMre-rankingspanproposalvideo-languagegrounding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that zero-shot video moment retrieval fails when it relies on matching a text query to video content, because the two live in different representation spaces (modality gap) and because machine-written captions do not read like human queries (language-style gap). To sidestep both gaps, Self-SiMS proposes candidate moments purely from the video's internal temporal self-similarity — how similar each frame is to its neighbors, computed from both visual features and MLLM-generated captions — and only then scores those candidates against the query. Scoring combines a top-k query–caption match inside each span with a self-matching score that rewards internally consistent segments, and a final MLLM yes/no verification re-ranks the top candidates. The paper reports state-of-the-art results on QVHighlights (R1@0.5 of 59.7 vs 58.3 for Moment-GPT, mAP@avg of 38.3 vs 35.0) and argues that query-independent proposal generation is why. If correct, this shifts the burden in zero-shot localization from cross-modal matching to intra-video structure, with a cheap query-aware verification on top.

What carries the argument

The central mechanism is the Temporal Self-similarity Matrix (TSM): an L×L matrix M = ½(Mf + Mc) where Mf and Mc are Gram matrices of ℓ2-normalized frame (DINOv2) and caption (LLaMA-3.2-Vision) features. A contrastive kernel K, with positive weights on the two diagonal quadrants and negative weights on the off-diagonal quadrants, is applied to local patches around the diagonal to produce a boundary score per frame (Eq. 1). Boundaries are selected as local maxima above instance-wise dynamic thresholds τj = 1 − λj V, where V is the ℓ2-mean of zero-centred frame features, so videos with more diverse contexts get finer partitions. Candidate spans are then scored by combining the Query-Matching Span Score (top-kS query–caption similarities) with the Self-Matching Span Score (mean TSM similarity between the key frame and the whole span), and finally re-ranked by an MLLM Yes/No verification on sampled frames. The key design decision is that the proposal stage never touches the query, which is what avoids the two gaps.

What would settle it

Find a benchmark or construct a video set where a ground-truth moment contains a strong self-similarity boundary in its interior (for instance, a mid-event camera cut or scene change, as in 'the whole press conference' when the camera switches angles). Since Self-SiMS generates non-overlapping spans from boundaries, the full ground-truth interval will never appear among the candidates; if such moments are common, its recall at high IoU should collapse even though the scoring and re-ranking stages are untouched.

Watch

Extended reading notes

Core claim

The central claim is that the modality and language-style gaps that plague zero-shot video moment retrieval can be largely bypassed by generating and scoring candidate spans from intra-video self-similarity rather than from query-to-video similarity. Self-SiMS builds two temporal self-similarity matrices — one from frame features, one from MLLM caption features — merges them, and detects event boundaries with a contrastive kernel, so candidate spans are defined by the video's own temporal coherence. Each span is scored by a weighted mix of a query-matching score (average of the top-kS query–caption similarities in the span) and a self-matching score (similarity between the span's key frame and all its frames), then the top-kC spans are re-ranked by an MLLM that answers whether the query is true of sampled representative frames. The paper demonstrates on QVHighlights that this ordering — query-agnostic proposal, query-aware re-verification — outperforms both query–frame similarity (TFVTG) and query–caption similarity (Moment-GPT) baselines, and presents the Inner-to-Outer Ratio as evidence that self-similarity separates true moments from surrounding context more sharply than query-based similarities.

Load-bearing premise

The method collapses if changes in a video's internal self-similarity do not align with the semantic events that text queries refer to, because the candidate spans are generated before the query is ever considered and no later scoring can fix a span that was never proposed.

Editorial extensions

If this is right

Query-agnostic proposal generation plus query-aware re-verification becomes a viable template for zero-shot retrieval, reducing reliance on fragile cross-modal similarity.
On QVHighlights, the method's reported R1@0.7 of 42.2 vs 37.7 for Moment-GPT suggests that self-similarity boundaries yield proposals that line up with the ground truth under strict IoU.
The dynamic threshold mechanism removes per-dataset manual threshold tuning, since the threshold adapts to intra-video context variability.
The MLLM re-ranking stage improves results under the same captioning backbone (MiniGPT-v2) as Moment-GPT, indicating the gains are not merely from a larger model.
The Inner-to-Outer Ratio diagnostic can be used to audit any similarity-based proposal method for whether it separates relevant from irrelevant segments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Self-similarity-based proposal generation could transfer to other modality-gap settings, such as audio-visual moment retrieval or image-text grounding, wherever intra-domain temporal structure exists.
The boundary-scoring module could be reused as a drop-in proposal generator for other retrieval and segmentation pipelines, independent of the query.
A testable extension would be to replace the single key frame with a small set of key frames when computing the Self-Matching Span Score, which might improve robustness for spans with multiple sub-events.
The IOR metric could serve as a dataset-level diagnostic to predict which videos or queries will be hard for query-based methods before running a full retrieval experiment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a decent empirical paper. The core idea—replace query-based similarity with intra-video self-similarity for span generation and scoring in training-free zero-shot moment retrieval—is not entirely new (it borrows UBoCo's boundary kernel and the pseudo-proposal idea from PSVL/LF-ZSVG), but the application to training-free ZMR, the top-k query-caption scoring, and the MLLM re-ranking together form a coherent system. The paper does what a good empirical paper should: it identifies two failure modes (modality gap and language-style gap), quantifies them with an IOR distribution, and then shows component ablations that support the design. The MLLM-choice ablation in the supplement is especially creditable: swapping in MiniGPT-v2, the same backbone Moment-GPT uses, still gives competitive or better numbers, so the gain isn't just a bigger model. I believe the results.

The main weakness is the "state-of-the-art across ZMR benchmarks" claim. On QVHighlights and TVR the method clearly leads, but on ActivityNet-Captions Moment-GPT gets better R1@0.5 (31.1 vs 28.2) and R1@0.7 (14.9 vs 13.8); on Charades-STA the R1@0.7 is 21.0 vs 21.6. The mIoU is better on both, so it's not a washout, but "state-of-the-art across" is simply not what Table 2 shows. The abstract should be tuned to "state-of-the-art on QVHighlights" or "competitive on others."

The stress-test note about oracle recall is fair and worth taking seriously. The whole pipeline stands on the query-agnostic proposal stage: if self-similarity boundaries don't cover the ground truth, the later scoring and re-ranking cannot recover. The paper reports only Oracle mIoU (71.13 on QVHighlights), which can hide a long tail of videos where no candidate overlaps the target. Because the final R1@0.5 on QVHighlights is 61.0, oracle recall there must be at least that high, so the argument survives on the flagship benchmark. But on Charades-STA and ActivityNet we don't have the ceiling number, and the lower downstream recall makes it worth checking. This is a missing experiment, not a fatal flaw. Also missing: confidence intervals or multiple runs, and code. Minor: the kernel size NK is never specified; a few other details live only in the supplement.

Who should read this: anyone working on zero-shot video grounding or training-free MLLM pipelines. It deserves a serious referee; with a corrected SOTA claim, an oracle-recall plot, and code, it would be a solid acceptance. I'd engage with it.

Referee Report

2 major / 6 minor

Summary. The paper proposes Self-SiMS, a training-free zero-shot video moment retrieval method. Instead of relying on query-to-video similarity for candidate generation, it builds temporal self-similarity maps from frame features and MLLM-generated caption features, detects boundaries with a contrastive kernel, and generates candidate spans. Each span is scored by combining a top-k Query-Matching Span Score (QMS) and a Self-Matching Span Score (SMS), and the top few spans are re-ranked by an MLLM using a yes/no verification. Experiments are reported on QVHighlights, Charades-STA, ActivityNet-Captions, and TVR, with the paper claiming state-of-the-art performance across ZMR benchmarks.

Significance. If the results hold, the paper makes a useful and reasonably original contribution: it provides a training-free alternative to query-dependent proposal generation, and the IOR diagnostic is a concrete way to expose modality and language-style gaps. The component ablations in Table 5 and the supplementary MLLM/backbone ablations (Tables 8-11) are careful and support the conclusion that QMS, SMS, and re-ranking each contribute, and that the gains are not simply due to switching to a larger MLLM. The main weaknesses are that the state-of-the-art claim is broader than the reported results justify, and that the query-agnostic proposal stage is validated only with Oracle mIoU rather than with a recall-ceiling analysis. Both issues are fixable within the scope of a revision.

major comments (2)

[Abstract; Contribution 4; Tables 2-3] The abstract and Contribution 4 claim 'state-of-the-art performance on ZMR benchmarks,' but Table 2 does not support this across all benchmarks. On ActivityNet-Captions, Self-SiMS trails Moment-GPT at R1@0.5 (28.2 vs 31.1) and R1@0.7 (13.8 vs 14.9); on Charades-STA it trails Moment-GPT at R1@0.7 (21.0 vs 21.6) and trails TFVTG at R1@0.3 (62.7 vs 64.8). The claim should be restricted to QVHighlights and TVR, or to mIoU on the other datasets, with an explicit discussion of the recall deficits.
[Sec. 3.1, Eqs. (2)-(4); Sec. 4.3, Table 4] The proposal stage is query-agnostic, and the later scoring stages in Eqs. (6), (8), and (10) can only re-rank spans already present in the candidate set. The paper reports only Oracle mIoU in Table 4, which is an average and can mask a long tail of videos where no candidate overlaps the ground truth. Please report oracle Recall@1 at IoU 0.5 and 0.7 on QVHighlights validation, and ideally the same statistic on Charades-STA and ActivityNet-Captions, to establish that the self-similarity boundary detector provides sufficient coverage for the later query-aware stages to succeed.

minor comments (6)

[Eq. (1)] As written, Eq. (1) defines b_i as the element-wise product P_i ⊙ K, which is a matrix rather than a scalar; the intended operation is presumably the sum of the element-wise products. In addition, the kernel size N_K is never specified anywhere in the main text or appendix.
[Sec. 2, Fig. 2] The Inner-to-Outer Ratio (IOR) is used to support the central gap-diagnosis claim, but the paper never gives an exact formula; please specify how the inner and outer regions are defined for query-frame similarity, query-caption similarity, and the self-similarity map.
[Sec. 3.1 and Eq. (5)] The frame encoder and the text/caption encoder used to compute F^f, F^c, and F^q are not identified in the main text; the supplementary ablations imply DINOv2 for frames and Sentence-BERT-type encoders for text, but the exact models, the captioning prompt, and the mapping from captions to F^c should be stated explicitly.
[Sec. 7.3, Table 10] On QVHighlights validation, the MiniGPT-v2 captioner/re-ranker variant outperforms the default LLaMA-3.2-Vision configuration (mAP 40.0 vs 39.3, R1@0.5 65.0 vs 61.0), yet the main paper reports the LLaMA version; please discuss this discrepancy and justify the default choice.
[Table 3] The TVR comparison omits Moment-GPT because the official implementation is unavailable, but the abstract's state-of-the-art claim should then be phrased so that readers understand the TVR result is a comparison against TFVTG only.
[Fig. 4] The span-length histograms in Fig. 4 are difficult to compare because the y-axis scales differ and the counts are unnormalized; normalized distributions or overlaid density curves would better support the qualitative comparison with the ground-truth length distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical pipeline with validation-tuned hyperparameters and held-out benchmark evaluation; no derivation reduces to its inputs.

full rationale

The paper's central claim is an empirical state-of-the-art result on held-out ZMR benchmarks, so the main correctness question is whether the method's components are fitted to the test labels; they are not. Candidate spans come from a query-agnostic self-similarity boundary detector (Eqs. 1-4) using a contrastive kernel adopted from external work [12]; no parameter of that detector is fitted to retrieval targets. The scoring stage combines top-k query-caption similarities (Eq. 6), a self-matching score from the same temporal self-similarity matrix (Eq. 8), and MLLM yes/no re-ranking (Eq. 10); hyperparameters alpha, beta, kS, kC, and NR are set by validation sweeps reported in the supplement, which is standard tuning rather than disguised prediction. The IOR distributions and Oracle mIoU are diagnostic analyses computed with ground-truth spans, not outputs of the method, so they cannot be 'predicted by construction.' The only author-overlap citation we can identify is [13] in Related Work, and it is used to contrast the training-based prior art rather than to justify a load-bearing premise; because it is an externally published, falsifiable result, it does not raise the circularity score. The missing oracle-recall check is an evidence gap about proposal ceiling, not a circularity of the derivation.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical method result. It depends on standard pretrained models, a set of hand-set hyperparameters (alpha, beta, kS, kR, kC, lambda values, NR bounds), and the assumption that self-similarity boundaries align with query-relevant moments. No new physical or mathematical entities are introduced; the IOR metric and severe LS-gap subset are analytic tools, not invented quantities with independent falsifiable handles.

free parameters (9)

alpha = 0.1
Blend of QMS and SMS in Eq. (9); chosen on validation ablations.
beta = 0.5
Blend of initial score and MLLM rerank score in Eq. (10); chosen for stability across datasets.
kS = 3
Top-k caption similarities in Eq. (6); ablations show kS=2-4 performs best.
kR = 3
Top-k MLLM rerank scores in Eq. (18) of the appendix; set to match kS.
kC = 5
Number of spans passed to MLLM reranking; balances efficiency and coverage.
Lambda threshold scaling = {1,2,3}
Constants in the dynamic threshold formula, Eq. (13) in the appendix; hand-set across datasets.
NR sampling bounds = rate 0.5, min 10, max 30
Number of frames sampled per span for MLLM reranking; main text says half the span length clamped to these bounds.
fusion weight for self-similarity maps = 0.5
Equal weighting of frame and caption self-similarity maps in M = 0.5(Mf + Mc); no ablation reported for this weight.
boundary kernel size NK = unspecified
Size of the contrastive kernel K in Eq. (1), referenced to UBoCo without an explicit value in this paper.

assumptions (4)

domain assumption Pretrained DINOv2 features and LLaMA caption embeddings are semantically comparable via cosine similarity.
Section 3.1 uses l2-normalized features and dot products to build self-similarity maps; if the embedding space is not semantically smooth, the boundary scores are meaningless.
domain assumption Self-similarity boundaries from the contrastive kernel correspond to event boundaries that contain query-relevant moments.
Section 3.2 generates all candidate spans from boundary scores, before any query signal is introduced. If this assumption fails, the rest of the pipeline cannot recover the correct span.
domain assumption MLLM yes/no logits are calibrated enough to rank query-frame relevance.
Section 3.3 softmaxes Yes/No logits and uses the Yes probability as a span re-ranking score; miscalibrated logits would distort the final ranking.
domain assumption Benchmark annotations in QVHighlights, Charades-STA, ActivityNet-Captions, and TVR are valid ground truth for moment retrieval.
All performance claims are evaluated against these datasets; noise or ambiguity in the annotations would directly affect the reported numbers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval." pith.science (2026). https://pith.science/paper/2URLOLOP

@misc{pith2026260719027,
  author       = {Pith},
  title        = {Pith review of: Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2URLOLOP}},
  note         = {Machine review of arXiv:2607.19027}
}

read the original abstract

Zero-shot video moment retrieval aims to overcome the limitations of traditional approaches that require large-scale datasets annotated with text and its relevant temporal spans. Despite advances in pre-trained vision-language models and multimodal large language models, existing ZMR methods still heavily depend on query-to-video content similarity, making them vulnerable to modality and language-style gaps. These gaps lead to unreliable span proposals and unstable moment retrieval results. To address this issue, we propose Self-Similarity-based Moment Proposal and Scoring that instead exploits intrinsic relationships within videos, enabling robust span generation and scoring. By deriving self-similarity only from the video content, we circumvent the noisy and mismatched patterns of query-frame or query-caption similarities, thereby mitigating both modality and language-style gaps. Furthermore, we introduce a query-aware MLLM-based reasoning stage to further sharpen alignment between text and video. Extensive experiments demonstrate that Self-SiMS achieves state-of-the-art performance across ZMR benchmarks.

Figures

Figures reproduced from arXiv: 2607.19027 by the authors.

**Figure 1.** Comparison of different similarity computation for candidate span generation. (a) Query–Frame similarity suffers from a modality gap between textual and visual features, producing low and flat similarity signals. (b) Query–Caption similarity alleviates modality issues but introduces a language-style gap, yielding noisy and inconsistent similarity patterns. (c) Our method leverages intra-video self-similarity, avoid… view at source ↗

**Figure 2.** Illustration and comparison of the Inner-to-Outer Ratio (IOR). (a) The IOR measures how much higher the similarity is within the ground-truth (Inner) region than in the surrounding (Outer) area. The middle and right examples illustrate how the Inner and Outer regions are obtained from query-based similarity scores and from the selfsimilarity map, respectively. (b) Comparison of IOR distributions on the QVHighlights… view at source ↗

**Figure 3.** The overview of our Self-Similarity based Moment proposal and Scoring (SelfSiMS). First, features from the frames and captions are extracted to construct two distinct self-similarity maps, Mf and Mc , which are then aggregated into a unified map, M. Each frame is then evaluated with a boundary score computed using a contrastive kernel, and candidate spans are generated in temporal order (Sec. 3.2). Next, the spans … view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Histogram of all span lengths on Charades-STA (left) and ActivityNet-Captions (right). Method Oracle-mIoU (%) Avg. # Span Cand. TFVTG 38.16 8.38 Moment-GPT 65.01 7.62 Self-SiMS (Ours) 71.13 6.46 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Inner-to-Outer Ratio (IOR) distributions on Charades (left) and ActivityNetCaptions (right). differences observed on these two datasets, particularly the recall tendencies in Tab. 2. TFVTG, affected by the modality gap, produces uniform similarity scores with weak te…

**Figure 6.** Figure 6: Qualitative results on QVHighlights (left) and Charades-STA (right). setting, the captioning cost is identical to that of Moment-GPT, while the reranking stage is slightly faster than when using LLaMA-3.2-Vision. Detailed performance results obtained with MiniGPT-v2 …

**Figure 7.** Figure 7: mIoU performance comparison between fixed thresholds and our dynamic threshold across three datasets. fixed-threshold setting is also evaluated as a set of three thresholds, matching the structure of our dynamic threshold set T . Across all datasets, the dynamic thresh…

**Figure 8.** Figure 8: Effect of weighting factor α on mIoU and recall across three datasets. Reranking score weighting factor β. For the final span score, the weighting factor β controls the contribution of the rereanked span score S R (m) relative to the initial span score S(m) [PITH_FUL…

**Figure 9.** Figure 9: Effect of weighting factor β on mIoU for QVHighlights, Charades-STA, and ActivityNet-Captions. QVHighlight Charades ActivityNet Analysis of weighting factor Top-𝑘𝑆 on mIoU performance 55 50 45 1 35 40 2 3 4 5 6 7 8 9 10 Our 𝑘𝑆 value [PITH_FULL_IMAGE:figures/full_fig_p…

**Figure 10.** Figure 10: Effect of Top-kS selection on mIoU for QVHighlights, Charades-STA, and ActivityNet-Captions. choice of kS. Ablation on the self-similarity source for span generation. Tab. 7 compares three variants of the temporal self-similarity map source used for span generation:…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 29 canonical work pages

[1]

arXiv preprint arXiv:2303.08774 (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[2]

IEEE transactions on pattern analysis and machine intelligence38(10), 2069–2081 (2015)

Barrett, D.P., Barbu, A., Siddharth, N., Siskind, J.M.: Saying what you’re looking for: Linguistics meets video search. IEEE transactions on pattern analysis and machine intelligence38(10), 2069–2081 (2015)

work page 2015
[3]

In: Proceedings of the ieee conference on computer vision and pattern recognition

Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the ieee conference on computer vision and pattern recognition. pp. 961–970 (2015)

2015
[4]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Chatterjee,D.,Remelli,E.,Song,Y.,Tekin,B.,Mittal,A.,Bhatnagar,B.,Camgoz, N.C., Hampali, S., Sauser, E., Ma, S., et al.: Streaming videollms for real-time procedural video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22586–22598 (2025)

work page 2025
[5]

arXiv preprint arXiv:2310.09478 (2023)

Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chan- dra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)

arXiv 2023
[6]

arXiv preprint arXiv:2407.21439 (2024)

Chen, Z., Xu, C., Qi, Y., Guo, J.: Mllm is a strong reranker: Advancing mul- timodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training. arXiv preprint arXiv:2407.21439 (2024)

arXiv 2024
[7]

In: Transfer Learning for Natural Language Processing Workshop

Diwan, A., Peng, P., Mooney, R.: Zero-shot video moment retrieval with off-the- shelf models. In: Transfer Learning for Natural Language Processing Workshop. pp. 10–21. PMLR (2023)

work page 2023
[8]

In: Proceedings of the IEEE international conference on computer vision

Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: Temporal activity localization via language query. In: Proceedings of the IEEE international conference on computer vision. pp. 5267–5275 (2017)

2017

Show all 52 references

[9]

arXiv preprint arXiv:2407.21783 (2024)

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

2024 arXiv
[10]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Huang, B., Wang, X., Chen, H., Song, Z., Zhu, W.: Vtimellm: Empower llm to grasp video moments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14271–14280 (2024)

2024
[11]

In: Companion Proceedings of the ACM on Web Conference 2025

Jin, C., Peng, H., Zhao, S., Wang, Z., Xu, W., Han, L., Zhao, J., Zhong, K., Rajasekaran, S., Metaxas, D.N.: Apeer: Automatic prompt engineering enhances large language model reranking. In: Companion Proceedings of the ACM on Web Conference 2025. pp. 2494–2502 (2025)

2025
[12]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Kang, H., Kim, J., Kim, T., Kim, S.J.: Uboco: Unsupervised boundary contrastive learning for generic event boundary detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20073–20082 (2022)

2022
[13]

In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

Kim, D., Park, J., Lee, J., Park, S., Sohn, K.: Language-free training for zero- shot video grounding. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2539–2548 (2023)

2023
[14]

In: Proceedings of the 31st ACM International Conference on Multimedia

Kong, S., Li, L., Zhang, B., Wang, W., Jiang, B., Yan, C., Xu, C.: Dynamic contrastive learning with pseudo-samples intervention for weakly supervised joint video mr and hd. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 538–546 (2023)

2023
[15]

In: Proceedings of the IEEE international conference on computer vision

Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Carlos Niebles, J.: Dense-captioning events in videos. In: Proceedings of the IEEE international conference on computer vision. pp. 706–715 (2017) 16 J. Lee et al

2017
[16]

Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catanzaro, B., Ping, W.: Nv-embed: Improved techniques for training llms as generalist embedding models (2025),https://arxiv.org/abs/2405.17428

2025 arXiv
[17]

Advances in Neural Information Processing Systems34, 11846–11858 (2021)

Lei, J., Berg, T.L., Bansal, M.: Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems34, 11846–11858 (2021)

2021
[18]

In: Computer Vision–ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16

Lei, J., Yu, L., Berg, T.L., Bansal, M.: Tvr: A large-scale dataset for video- subtitle moment retrieval. In: Computer Vision–ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. pp. 447–463. Springer (2020)

2020
[19]

In: International conference on machine learning

Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

2023
[20]

In: International confer- ence on machine learning

Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International confer- ence on machine learning. pp. 12888–12900. PMLR (2022)

2022
[21]

arXiv preprint arXiv:2411.02571 (2024)

Lin, S.C., Lee, C., Shoeybi, M., Lin, J., Catanzaro, B., Ping, W.: Mm-embed: Uni- versal multimodal retrieval with multimodal llms. arXiv preprint arXiv:2411.02571 (2024)

2024 arXiv
[22]

Liu, D., Qu, X., Dong, J., Nan, G., Zhou, P., Xu, Z., Chen, L., Yan, H., Cheng, Y.: Filling the information gap between video and query for language-driven moment retrieval.In:Proceedingsofthe31stACMInternationalConferenceonMultimedia. pp. 4190–4199 (2023)

2023
[23]

In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

Liu, Y., Li, S., Wu, Y., Chen, C.W., Shan, Y., Qie, X.: Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3042–3051 (2022)

2022
[24]

In: Proceedings of the AAAI conference on artificial intelligence

Liu,Z.,Li,J.,Xie,H.,Li,P.,Ge,J.,Liu,S.A.,Jin,G.:Towardsbalancedalignment: Modal-enhanced semantic modeling for video moment retrieval. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 3855–3863 (2024)

2024
[25]

In: Proceedings of the IEEE/CVF winter con- ference on applications of computer vision

Luo, D., Huang, J., Gong, S., Jin, H., Liu, Y.: Zero-shot video moment retrieval from frozen vision-language models. In: Proceedings of the IEEE/CVF winter con- ference on applications of computer vision. pp. 5464–5473 (2024)

2024
[26]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Ma, K., Zang, X., Feng, Z., Fang, H., Ban, C., Wei, Y., He, Z., Li, Y., Sun, H.: Llavilo: Boosting video moment retrieval via adapter-based multimodal modeling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2798–2803 (2023)

2023
[27]

In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024)

2024
[28]

arXiv preprint arXiv:2306.05424 (2023)

Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards de- tailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

2023 arXiv
[29]

In: Proceedings of the IEEE/CVF international conference on computer vision

Nam, J., Ahn, D., Kang, D., Ha, S.J., Choi, J.: Zero-shot natural language video lo- calization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1470–1479 (2021)

2021
[30]

arXiv preprint arXiv:2304.07193 (2023) Mitigating Modality and Language-Style Gaps for ZMR 17

Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) Mitigating Modality and Language-Style G...

2023 arXiv
[31]

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021),https://arxiv.org/abs/ 2103.00020

2021 arXiv
[32]

In: International conference on machine learning

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

2021
[33]

arXiv preprint arXiv:1908.10084 (2019)

Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.10084 (2019)

2019 arXiv
[34]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Ren, S., Yao, L., Li, S., Sun, X., Hou, L.: Timechat: A time-sensitive multi- modal large language model for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14313– 14323 (2024)

2024
[35]

In: 2007 IEEE conference on computer vision and pattern recognition

Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: 2007 IEEE conference on computer vision and pattern recognition. pp. 1–8. IEEE (2007)

2007
[36]

arXiv preprint arXiv:2304.09542 (2023)

Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., Ren, Z.: Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542 (2023)

2023 arXiv
[37]

IEEE Signal Process- ing Letters31, 521–525 (2023)

Sun, Y., Xu, Y., Xie, Z., Shu, Y., Du, S.: Gptsee: Enhancing moment retrieval and highlight detection via description-based similarity features. IEEE Signal Process- ing Letters31, 521–525 (2023)

2023
[38]

arXiv preprint arXiv:2302.13971 (2023)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

2023 arXiv
[39]

arXiv preprint arXiv:2307.09288 (2023)

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

2023 arXiv
[40]

In: Proceedings of the 30th ACM international conference on multimedia

Wang,G.,Wu,X.,Liu,Z.,Yan,J.:Prompt-basedzero-shotvideomomentretrieval. In: Proceedings of the 30th ACM international conference on multimedia. pp. 413– 421 (2022)

2022
[41]

Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Improving text embeddings with large language models (2024),https://arxiv.org/abs/2401. 00368

2024
[42]

Wattasseril, J.I., Shekhar, S., Döllner, J., Trapp, M.: Zero-shot video moment re- trievalusingblip-basedmodels.In:InternationalSymposiumonVisualComputing. pp. 160–171. Springer (2023)

2023
[43]

arXiv preprint arXiv:2505.12499 (2025)

Xiao, J., Song, Z., Hu, J., Cheng, H., Hu, Z., Li, J., Hong, R.: Contrastive align- ment with semantic gap-aware corrections in text-video retrieval. arXiv preprint arXiv:2505.12499 (2025)

2025
[44]

Applied Sciences14(5), 1894 (Feb 2024).https: //doi.org/10.3390/app14051894,http://dx.doi.org/10.3390/app14051894

Xu, Y., Sun, Y., Xie, Z., Zhai, B., Du, S.: Vtg-gpt: Tuning-free zero-shot video temporal grounding with gpt. Applied Sciences14(5), 1894 (Feb 2024).https: //doi.org/10.3390/app14051894,http://dx.doi.org/10.3390/app14051894

2024 doi
[45]

In: 2024 International Joint Conference on Neural Networks (IJCNN)

Xu, Y., Sun, Y., Zhai, B., Jia, Y., Du, S.: Mh-detr: Video moment and highlight detection with cross-modal transformer. In: 2024 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2024)

2024
[46]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Xu, Y., Sun, Y., Zhai, B., Li, M., Liang, W., Li, Y., Du, S.: Zero-shot video moment retrieval via off-the-shelf multimodal large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 8978–8986 (2025) 18 J. Lee et al

2025
[47]

In: 2024 IEEE International Conference on Multimedia and Expo (ICME)

Xu, Y., Sun, Y., Zhai, B., Xie, Z., Jia, Y., Du, S.: Multi-modal fusion and query refinement network for video moment retrieval and highlight detection. In: 2024 IEEE International Conference on Multimedia and Expo (ICME). pp. 1–6. IEEE (2024)

2024
[48]

Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training (2023)

2023
[49]

arXiv preprint arXiv:2306.02858 (2023)

Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

2023 arXiv
[50]

In: European Conference on Com- puter Vision

Zheng, M., Cai, X., Chen, Q., Peng, Y., Liu, Y.: Training-free video temporal grounding using large-scale pre-trained models. In: European Conference on Com- puter Vision. pp. 20–37. Springer (2024)

2024
[51]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Zheng, M., Huang, Y., Chen, Q., Liu, Y.: Weakly supervised video moment lo- calization with contrastive negative sample mining. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 3517–3525 (2022)

2022
[52]

Yes” and “No

Zheng, M., Huang, Y., Chen, Q., Peng, Y., Liu, Y.: Weakly supervised temporal sentence grounding with gaussian-based contrastive proposal learning. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15555–15564 (2022) Mitigating Modali...

2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2303.08774 (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[2] [2]

IEEE transactions on pattern analysis and machine intelligence38(10), 2069–2081 (2015)

Barrett, D.P., Barbu, A., Siddharth, N., Siskind, J.M.: Saying what you’re looking for: Linguistics meets video search. IEEE transactions on pattern analysis and machine intelligence38(10), 2069–2081 (2015)

work page 2015

[3] [3]

In: Proceedings of the ieee conference on computer vision and pattern recognition

Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the ieee conference on computer vision and pattern recognition. pp. 961–970 (2015)

2015

[4] [4]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Chatterjee,D.,Remelli,E.,Song,Y.,Tekin,B.,Mittal,A.,Bhatnagar,B.,Camgoz, N.C., Hampali, S., Sauser, E., Ma, S., et al.: Streaming videollms for real-time procedural video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22586–22598 (2025)

work page 2025

[5] [5]

arXiv preprint arXiv:2310.09478 (2023)

Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chan- dra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)

arXiv 2023

[6] [6]

arXiv preprint arXiv:2407.21439 (2024)

Chen, Z., Xu, C., Qi, Y., Guo, J.: Mllm is a strong reranker: Advancing mul- timodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training. arXiv preprint arXiv:2407.21439 (2024)

arXiv 2024

[7] [7]

In: Transfer Learning for Natural Language Processing Workshop

Diwan, A., Peng, P., Mooney, R.: Zero-shot video moment retrieval with off-the- shelf models. In: Transfer Learning for Natural Language Processing Workshop. pp. 10–21. PMLR (2023)

work page 2023

[8] [8]

In: Proceedings of the IEEE international conference on computer vision

Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: Temporal activity localization via language query. In: Proceedings of the IEEE international conference on computer vision. pp. 5267–5275 (2017)

2017