{"id":"a9ca563b-49a9-45d3-8c97-5f6e4058d5fe","arxiv_id":"2607.19027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Self-SiMS generates video moment candidates from intra-video self-similarity and re-ranks them with an MLLM, reporting state-of-the-art numbers on QVHighlights and mixed results on other benchmarks.","lead":"This paper introduces Self-SiMS, a training-free method for finding video moments from a text query. Instead of matching the query to every frame, it uses changes in how similar frames are to each other to propose candidate moments, then re-ranks them with a vision-language model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Query-agnostic proposal stage lacks an oracle-recall check; if candidate spans miss the ground truth, later scoring cannot recover, yet the paper reports only Oracle mIoU.","rationale":"I agree with the reader that the weakest assumption is the query-agnostic self-similarity proposal. My concern sharpens this: the missing evidence is not Oracle mIoU but oracle recall at the actual evaluation IoU thresholds. Oracle mIoU can be inflated by a few high-overlap candidates while a large fraction of queries have no candidate with IoU >= 0.5; those queries are unrecoverable. The paper's own Table 4 shows Oracle mIoU 71.13 with only 6.46 candidates per video, which is promising, but mIoU is not a recall measure. The ablation in Table 13 shows self-similarity proposals beat sliding windows and k-means, and the IOR analysis supports the intuition, so the concern is not that the approach is wrong; it is that the decisive statistic is unmeasured. The overstatement of SOTA on ActivityNet (Table 2, R1@0.5 and R1@0.7 trail Moment-GPT) is a real secondary issue but does not invalidate the method; it only weakens the 'across benchmarks' phrasing. If the oracle-recall check shows high recall, the reader's CONDITIONAL verdict can stand with a revised SOTA claim; if not, the method's central assumption fails for a substantial query subset and the claim should be further restricted. I recommend no verdict change beyond the reader's CONDITIONAL.","tokens_in":20879,"tokens_out":8838,"duration_ms":73903,"concrete_test":"On QVHighlights validation, Charades-STA, and ActivityNet-Captions, compute oracle Recall@1 at IoU 0.5 and 0.7 by taking, for each video, the candidate span in E with maximum IoU against the ground-truth span and treating that as the prediction. If oracle R1@0.5 is materially higher than the final R1@0.5 (e.g., above 75 on QVHighlights where final is 61), then the proposal stage is not the bottleneck and the central assumption holds; if oracle R1@0.5 is close to or below the final R1@0.5, or is low in absolute terms, the query-agnostic proposal ceiling is the limiting factor and the SOTA claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Self-SiMS achieves state-of-the-art ZMR performance rests on the query-agnostic candidate span generator (Sec. 3.1, Eqs. 1-4): the later query-matching score (Eq. 6), self-matching score (Eq. 8), and MLLM re-ranking (Sec. 3.3) can only select among spans already present in E. If the ground-truth moment is not covered by any candidate span with sufficient IoU, no subsequent stage can retrieve it. The paper's evidence for proposal quality is Oracle mIoU (Table 4, 71.13 on QVHighlights val), but Oracle mIoU averages the best IoU over candidates and can mask a long tail of videos where all candidates miss the target. The paper does not report oracle Recall@1 at the evaluation thresholds (0.5/0.7), nor the recall ceiling on Charades-STA and ActivityNet-Captions. Since the whole method is built on the assumption that self-similarity boundaries track query-relevant semantic events, the missing oracle-recall statistic is the most load-bearing gap in the argument for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-SiMS, a training-free zero-shot video moment retrieval method. Instead of relying on query-to-video similarity for candidate generation, it builds temporal self-similarity maps from frame features and MLLM-generated caption features, detects boundaries with a contrastive kernel, and generates candidate spans. Each span is scored by combining a top-k Query-Matching Span Score (QMS) and a Self-Matching Span Score (SMS), and the top few spans are re-ranked by an MLLM using a yes/no verification. Experiments are reported on QVHighlights, Charades-STA, ActivityNet-Captions, and TVR, with the paper claiming state-of-the-art performance across ZMR benchmarks.","tokens_in":21177,"tokens_out":7701,"duration_ms":68719,"significance":"If the results hold, the paper makes a useful and reasonably original contribution: it provides a training-free alternative to query-dependent proposal generation, and the IOR diagnostic is a concrete way to expose modality and language-style gaps. The component ablations in Table 5 and the supplementary MLLM/backbone ablations (Tables 8-11) are careful and support the conclusion that QMS, SMS, and re-ranking each contribute, and that the gains are not simply due to switching to a larger MLLM. The main weaknesses are that the state-of-the-art claim is broader than the reported results justify, and that the query-agnostic proposal stage is validated only with Oracle mIoU rather than with a recall-ceiling analysis. Both issues are fixable within the scope of a revision.","major_comments":[{"comment":"The abstract and Contribution 4 claim 'state-of-the-art performance on ZMR benchmarks,' but Table 2 does not support this across all benchmarks. On ActivityNet-Captions, Self-SiMS trails Moment-GPT at R1@0.5 (28.2 vs 31.1) and R1@0.7 (13.8 vs 14.9); on Charades-STA it trails Moment-GPT at R1@0.7 (21.0 vs 21.6) and trails TFVTG at R1@0.3 (62.7 vs 64.8). The claim should be restricted to QVHighlights and TVR, or to mIoU on the other datasets, with an explicit discussion of the recall deficits.","section":"Abstract; Contribution 4; Tables 2-3"},{"comment":"The proposal stage is query-agnostic, and the later scoring stages in Eqs. (6), (8), and (10) can only re-rank spans already present in the candidate set. The paper reports only Oracle mIoU in Table 4, which is an average and can mask a long tail of videos where no candidate overlaps the ground truth. Please report oracle Recall@1 at IoU 0.5 and 0.7 on QVHighlights validation, and ideally the same statistic on Charades-STA and ActivityNet-Captions, to establish that the self-similarity boundary detector provides sufficient coverage for the later query-aware stages to succeed.","section":"Sec. 3.1, Eqs. (2)-(4); Sec. 4.3, Table 4"}],"minor_comments":[{"comment":"As written, Eq. (1) defines b_i as the element-wise product P_i ⊙ K, which is a matrix rather than a scalar; the intended operation is presumably the sum of the element-wise products. In addition, the kernel size N_K is never specified anywhere in the main text or appendix.","section":"Eq. (1)"},{"comment":"The Inner-to-Outer Ratio (IOR) is used to support the central gap-diagnosis claim, but the paper never gives an exact formula; please specify how the inner and outer regions are defined for query-frame similarity, query-caption similarity, and the self-similarity map.","section":"Sec. 2, Fig. 2"},{"comment":"The frame encoder and the text/caption encoder used to compute F^f, F^c, and F^q are not identified in the main text; the supplementary ablations imply DINOv2 for frames and Sentence-BERT-type encoders for text, but the exact models, the captioning prompt, and the mapping from captions to F^c should be stated explicitly.","section":"Sec. 3.1 and Eq. (5)"},{"comment":"On QVHighlights validation, the MiniGPT-v2 captioner/re-ranker variant outperforms the default LLaMA-3.2-Vision configuration (mAP 40.0 vs 39.3, R1@0.5 65.0 vs 61.0), yet the main paper reports the LLaMA version; please discuss this discrepancy and justify the default choice.","section":"Sec. 7.3, Table 10"},{"comment":"The TVR comparison omits Moment-GPT because the official implementation is unavailable, but the abstract's state-of-the-art claim should then be phrased so that readers understand the TVR result is a comparison against TFVTG only.","section":"Table 3"},{"comment":"The span-length histograms in Fig. 4 are difficult to compare because the y-axis scales differ and the counts are unnormalized; normalized distributions or overlaid density curves would better support the qualitative comparison with the ground-truth length distribution.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The two substantive issues are the overbroad state-of-the-art claim and the absence of an oracle-recall analysis for the query-agnostic proposal stage; both are addressable without changing the method. I would also ask the authors to add the missing implementation constants and the exact IOR formula, since the current supplementary material is helpful but not fully self-contained for reproduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a decent empirical paper. The core idea—replace query-based similarity with intra-video self-similarity for span generation and scoring in training-free zero-shot moment retrieval—is not entirely new (it borrows UBoCo's boundary kernel and the pseudo-proposal idea from PSVL/LF-ZSVG), but the application to training-free ZMR, the top-k query-caption scoring, and the MLLM re-ranking together form a coherent system. The paper does what a good empirical paper should: it identifies two failure modes (modality gap and language-style gap), quantifies them with an IOR distribution, and then shows component ablations that support the design. The MLLM-choice ablation in the supplement is especially creditable: swapping in MiniGPT-v2, the same backbone Moment-GPT uses, still gives competitive or better numbers, so the gain isn't just a bigger model. I believe the results.\n\nThe main weakness is the \"state-of-the-art across ZMR benchmarks\" claim. On QVHighlights and TVR the method clearly leads, but on ActivityNet-Captions Moment-GPT gets better R1@0.5 (31.1 vs 28.2) and R1@0.7 (14.9 vs 13.8); on Charades-STA the R1@0.7 is 21.0 vs 21.6. The mIoU is better on both, so it's not a washout, but \"state-of-the-art across\" is simply not what Table 2 shows. The abstract should be tuned to \"state-of-the-art on QVHighlights\" or \"competitive on others.\"\n\nThe stress-test note about oracle recall is fair and worth taking seriously. The whole pipeline stands on the query-agnostic proposal stage: if self-similarity boundaries don't cover the ground truth, the later scoring and re-ranking cannot recover. The paper reports only Oracle mIoU (71.13 on QVHighlights), which can hide a long tail of videos where no candidate overlaps the target. Because the final R1@0.5 on QVHighlights is 61.0, oracle recall there must be at least that high, so the argument survives on the flagship benchmark. But on Charades-STA and ActivityNet we don't have the ceiling number, and the lower downstream recall makes it worth checking. This is a missing experiment, not a fatal flaw. Also missing: confidence intervals or multiple runs, and code. Minor: the kernel size NK is never specified; a few other details live only in the supplement.\n\nWho should read this: anyone working on zero-shot video grounding or training-free MLLM pipelines. It deserves a serious referee; with a corrected SOTA claim, an oracle-recall plot, and code, it would be a solid acceptance. I'd engage with it.","headline":"Useful training-free ZMR method with honest ablations, but the SOTA claim overreaches outside QVHighlights and the query-agnostic proposal stage needs an oracle-recall check.","tokens_in":21696,"tokens_out":3066,"would_cite":false,"duration_ms":27514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot video moment retrieval works better when candidate moments are proposed from a video's own internal self-similarity, not from query-to-frame matching, and then verified by a multimodal language model.","keywords":["zero-shot video moment retrieval","self-similarity","temporal boundary detection","modality gap","language-style gap","MLLM re-ranking","span proposal","video-language grounding"],"falsifier":"Find a benchmark or construct a video set where a ground-truth moment contains a strong self-similarity boundary in its interior (for instance, a mid-event camera cut or scene change, as in 'the whole press conference' when the camera switches angles). Since Self-SiMS generates non-overlapping spans from boundaries, the full ground-truth interval will never appear among the candidates; if such moments are common, its recall at high IoU should collapse even though the scoring and re-ranking stages are untouched.","tokens_in":20726,"feed_emoji":"🎬","tokens_out":6837,"duration_ms":52411,"temperature":0.7,"pith_summary":"This paper argues that zero-shot video moment retrieval fails when it relies on matching a text query to video content, because the two live in different representation spaces (modality gap) and because machine-written captions do not read like human queries (language-style gap). To sidestep both gaps, Self-SiMS proposes candidate moments purely from the video's internal temporal self-similarity — how similar each frame is to its neighbors, computed from both visual features and MLLM-generated captions — and only then scores those candidates against the query. Scoring combines a top-k query–caption match inside each span with a self-matching score that rewards internally consistent segments, and a final MLLM yes/no verification re-ranks the top candidates. The paper reports state-of-the-art results on QVHighlights (R1@0.5 of 59.7 vs 58.3 for Moment-GPT, mAP@avg of 38.3 vs 35.0) and argues that query-independent proposal generation is why. If correct, this shifts the burden in zero-shot localization from cross-modal matching to intra-video structure, with a cheap query-aware verification on top.","feed_headline":"Self-similarity beats query matching for zero-shot moments","feed_subtitle":"Proposing spans from the video's own boundaries, then verifying with an MLLM, tops QVHighlights zero-shot methods.","key_machinery":"The central mechanism is the Temporal Self-similarity Matrix (TSM): an L×L matrix M = ½(Mf + Mc) where Mf and Mc are Gram matrices of ℓ2-normalized frame (DINOv2) and caption (LLaMA-3.2-Vision) features. A contrastive kernel K, with positive weights on the two diagonal quadrants and negative weights on the off-diagonal quadrants, is applied to local patches around the diagonal to produce a boundary score per frame (Eq. 1). Boundaries are selected as local maxima above instance-wise dynamic thresholds τj = 1 − λj V, where V is the ℓ2-mean of zero-centred frame features, so videos with more diverse contexts get finer partitions. Candidate spans are then scored by combining the Query-Matching Span Score (top-kS query–caption similarities) with the Self-Matching Span Score (mean TSM similarity between the key frame and the whole span), and finally re-ranked by an MLLM Yes/No verification on sampled frames. The key design decision is that the proposal stage never touches the query, which is what avoids the two gaps.","core_discovery":"The central claim is that the modality and language-style gaps that plague zero-shot video moment retrieval can be largely bypassed by generating and scoring candidate spans from intra-video self-similarity rather than from query-to-video similarity. Self-SiMS builds two temporal self-similarity matrices — one from frame features, one from MLLM caption features — merges them, and detects event boundaries with a contrastive kernel, so candidate spans are defined by the video's own temporal coherence. Each span is scored by a weighted mix of a query-matching score (average of the top-kS query–caption similarities in the span) and a self-matching score (similarity between the span's key frame and all its frames), then the top-kC spans are re-ranked by an MLLM that answers whether the query is true of sampled representative frames. The paper demonstrates on QVHighlights that this ordering — query-agnostic proposal, query-aware re-verification — outperforms both query–frame similarity (TFVTG) and query–caption similarity (Moment-GPT) baselines, and presents the Inner-to-Outer Ratio as evidence that self-similarity separates true moments from surrounding context more sharply than query-based similarities.","pith_inferences":["Self-similarity-based proposal generation could transfer to other modality-gap settings, such as audio-visual moment retrieval or image-text grounding, wherever intra-domain temporal structure exists.","The boundary-scoring module could be reused as a drop-in proposal generator for other retrieval and segmentation pipelines, independent of the query.","A testable extension would be to replace the single key frame with a small set of key frames when computing the Self-Matching Span Score, which might improve robustness for spans with multiple sub-events.","The IOR metric could serve as a dataset-level diagnostic to predict which videos or queries will be hard for query-based methods before running a full retrieval experiment."],"forward_implications":["Query-agnostic proposal generation plus query-aware re-verification becomes a viable template for zero-shot retrieval, reducing reliance on fragile cross-modal similarity.","On QVHighlights, the method's reported R1@0.7 of 42.2 vs 37.7 for Moment-GPT suggests that self-similarity boundaries yield proposals that line up with the ground truth under strict IoU.","The dynamic threshold mechanism removes per-dataset manual threshold tuning, since the threshold adapts to intra-video context variability.","The MLLM re-ranking stage improves results under the same captioning backbone (MiniGPT-v2) as Moment-GPT, indicating the gains are not merely from a larger model.","The Inner-to-Outer Ratio diagnostic can be used to audit any similarity-based proposal method for whether it separates relevant from irrelevant segments."],"supporting_citations":[{"why":"Supplies the event-boundary detection method and contrastive kernel that the boundary scoring is built on.","marker":"[12]"},{"why":"Baseline that uses query–frame similarity; its failure motivates the modality-gap argument.","marker":"[50]"},{"why":"Baseline that uses query–caption similarity; its noise motivates the language-style-gap argument and is the main comparison.","marker":"[46]"},{"why":"Visual encoder (DINOv2) whose frame features build the frame self-similarity matrix.","marker":"[30]"},{"why":"LLaMA-3.2-Vision, the MLLM used for captioning and the re-ranking stage.","marker":"[9]"},{"why":"MiniGPT-v2 used in ablations to show performance holds with the same captioning backbone as Moment-GPT.","marker":"[5]"}],"fun_headline_variants":["Self-similarity, not query matching, finds video moments","Zero-shot moments found via video's own structure","Intra-video similarity bypasses query gaps for moments","Propose moments from video self-similarity, not query","Self-SiMS: self-similarity beats query for zero-shot moments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method collapses if changes in a video's internal self-similarity do not align with the semantic events that text queries refer to, because the candidate spans are generated before the query is ever considered and no later scoring can fix a span that was never proposed.","fun_headline_variants_meta":{"raw":{"variants":["Self-similarity, not query matching, finds video moments","Zero-shot moments found via video's own structure","Intra-video similarity bypasses query gaps for moments","Propose moments from video self-similarity, not query","Self-SiMS: self-similarity beats query for zero-shot moments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000805,"raw_usage":{"total_tokens":3544,"prompt_tokens":962,"completion_tokens":2582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2498}},"tokens_in":578,"tokens_out":2582,"duration_ms":15952,"temperature":1.0,"reasoning_tokens":2498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:31:48.091745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a benchmark or construct a video set where a ground-truth moment contains a strong self-similarity boundary in its interior (for instance, a mid-event camera cut or scene change, as in 'the whole press conference' when the camera switches angles). Since Self-SiMS generates non-overlapping spans from boundaries, the full ground-truth interval will never appear among the candidates; if such moments are common, its recall at high IoU should collapse even though the scoring and re-ranking stages are untouched.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the event-boundary detection method and contrastive kernel that the boundary scoring is built on."},{"cited_title":"In: European Conference on Com- puter Vision","cited_arxiv_id":null,"evidence_quote":"Baseline that uses query–frame similarity; its failure motivates the modality-gap argument."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Baseline that uses query–caption similarity; its noise motivates the language-style-gap argument and is the main comparison."}],"review_version":2}