{"id":"5beee5ef-78d7-4e8c-bb64-6b64bf4f8a1f","arxiv_id":"2411.14505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLaVA-MR combines dense frame sampling, frame-difference key-frame selection, and variance-based token compression to improve generative MLLM video moment retrieval, reporting small SOTA gains over Mr. BLIP on three benchmarks.","lead":"This paper presents LLaVA-MR, a video moment retrieval model built on the BLIP-2 multimodal LLM, adding dense frame sampling with time encoding, key-frame selection, and token compression to locate query-relevant clips. The authors report state-of-the-art scores on Charades-STA, QVHighlights, and ActivityNet, but the supporting analysis contains a contradiction in the time encoding ablation and relies on test-set tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IFS's contribution is never isolated: Table 2 adds IFS and DTC together, Table 5 only varies compression, and no random key-frame control is reported, so the +2.13 R1@0.5 gain cannot be attributed to IFS.","rationale":"The paper reports strong benchmark numbers and a reasonable overall system, and the QVHighlights test-set results come from the official evaluation server, so the headline SOTA claim is not obviously false. The central novelty, however, is the combination of DFTE, IFS, and DTC, and the paper's ablations do not isolate IFS as a causal component. The +2.13 R1@0.5 gain attributed to 'IFS&DTC' is a joint change, and no control such as random key-frame selection is provided; the one qualitative illustration in Fig. 5 cannot establish the criterion's general validity. The baseline discrepancy between Table 2 (a) at 65.48 and Table 1's Mr. BLIP at 69.31 is a separate internal inconsistency that makes it harder to interpret the magnitude of the gains. Because the reader already returned CONDITIONAL largely on reproducibility and selective hyperparameter tuning, this concern reinforces that verdict without requiring a change: the SOTA numbers could still hold, but the mechanism story needs a controlled experiment before the three-module claim is accepted. If the random-selection control shows no drop, I should move to a less favorable verdict, but based on current evidence UNCHANGED is the appropriate recommendation.","tokens_in":16004,"tokens_out":11297,"duration_ms":106157,"concrete_test":"Run the full LLaVA-MR pipeline on Charades-STA with IFS's top-k selection (Eq. 3) replaced by random selection of k=32 key frames, keeping N=60, DTC T=16, LoRA, and the training recipe identical; repeat with at least 3 seeds and compare R1@0.5/mIoU against the IFS result (70.65/59.78). If random selection matches IFS within seed noise, the IFS ranking criterion is not load-bearing and the central three-module claim reduces to dense sampling plus time encoding plus compression.","verdict_should_be":"UNCHANGED","load_bearing_attack":"IFS is the paper's claimed mechanism for capturing brief visual and motion patterns, but its contribution is never isolated. Table 2 row (c)→(d) turns on IFS and DTC simultaneously, so the +2.13 R1@0.5 belongs to the pair, not to IFS. Table 5 then varies only the compression method (none vs average pooling vs variance-based DTC) with IFS fixed, so the selection criterion itself is never tested. The only qualitative support is one guitar video (Fig. 5). Moreover, the sparse baseline (a) in Table 2 scores 65.48 R1@0.5 on Charades-STA, while Table 1 reports Mr. BLIP at 69.31 on the same test set; the module gains are therefore measured from a roughly 3.8-point weaker baseline than the leading method the paper claims to surpass, inflating the apparent contribution of the proposed modules. With N, k, T, time encoding, and post-processing selected on the evaluation benchmarks and no error bars or multiple-seed results, the central claim that IFS's adjacent-L2-distance criterion is load-bearing is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaVA-MR, an MLLM-based video moment retrieval method built on BLIP-2. It introduces three modules: Dense Frame and Time Encoding (DFTE), which samples more frames and chooses either relative frame indices or timestamps as time tokens depending on the sampling rate; Informative Frame Selection (IFS), which selects top-k frames by smoothed adjacent-feature L2 distance; and Dynamic Token Compression (DTC), which compresses non-key frame tokens either by average pooling or by retaining high-variance Q-Former queries. The authors report state-of-the-art results on QVHighlights test (76.59 R1@0.5, 69.41 mAP@0.5), Charades-STA (70.65 R1@0.5), and ActivityNet Captions (55.16 R1@0.5), with ablations on Charades-STA showing monotone gains. The implementation is promised to be open-sourced upon acceptance.","tokens_in":16210,"tokens_out":7639,"duration_ms":69324,"significance":"If the reported results are reproducible, LLaVA-MR is a simple and potentially useful recipe for adapting frozen-image MLLMs to moment retrieval: dense sampling plus a hand-crafted frame-selection and token-compression scheme. A genuine strength is that the headline QVHighlights test numbers come from the evaluation server, so those labels were not directly visible to the authors. The paper also reports a modest inference-time advantage over Mr. BLIP. However, the specific mechanism claimed to be most novel, IFS, is not isolated in the ablations, and the experimental protocol contains enough selection-on-test-set and post-processing ambiguity that the central claim is not yet established to the standard expected for this venue. The work is nonetheless a reasonable engineering contribution that could become acceptable after substantial revision.","major_comments":[{"comment":"The contribution of IFS is never isolated. Row (c) to (d) of Table 2 enables IFS and DTC together, so the +2.13 R1@0.5 gain belongs to the combination, not to IFS alone. Table 5 then varies only the compression method (none, average pooling, variance-based DTC) with IFS fixed, so the selection criterion itself is not tested. Without an ablation that compares IFS against random key-frame selection, and against uniform/dense sampling at the same token budget, the paper's central claim that adjacent-feature L2 distance identifies informative frames is not supported. The interpretation is further complicated because the sparse baseline in Table 2 (a) scores 65.48 R1@0.5, well below Mr. BLIP's 69.31 on the same Charades-STA test set, so the module gains are measured from a weaker starting point than the leading baseline.","section":"§4.3, Tables 2 and 5"},{"comment":"For Charades-STA, the hyperparameters N=60 and k=32 are selected by optimizing performance on the Charades-STA test set, not on a held-out validation split. This is test-set selection and can inflate the reported gains. The QVHighlights hyperparameters are selected on the validation set (Table 8), but Charades-STA, where the full ablation and final SOTA comparison are run, is not handled this way. Please report a validation-based selection procedure for both datasets, or clearly state how many configurations were tried and account for the selection in the reported numbers.","section":"§4.3, Fig. 6 and Table 8"},{"comment":"The final reported numbers depend on post-processing heuristics described only qualitatively. The algorithm swaps start and end times, extracts number pairs from malformed outputs, and substitutes [-1,-1] for invalid predictions; such heuristics can change recall and mAP. The paper does not quantify the effect of post-processing on the final scores, nor does it state whether the Mr. BLIP baseline numbers were obtained with equivalent post-processing. This is necessary for a fair SOTA comparison, especially because the paper states that the heuristics are applied 'to improve prediction accuracy.'","section":"§4.1 and Supplementary §7"},{"comment":"The ablation tables are internally inconsistent. Table 1 reports LLaVA-MR at 78.13 R1@0.5 on QVHighlights validation, while Table 3 reports at most 76.06 for either time representation; Table 2 row (c) gives 68.52 R1@0.5 on Charades-STA, while Table 3 gives 67.83 or 66.37. The text also states that 'using frame indices as time tokens led to better performance' on QVHighlights, but Table 3 shows timestamps are better by 1.29 points. The configuration underlying each table (which modules, special tokens, and post-processing are active) must be stated, and the contradiction resolved, before the time-encoding design claim can be evaluated.","section":"§4.3, Tables 1 and 3"},{"comment":"All experiments appear to be single runs with no error bars. Given that the headline improvements over Mr. BLIP are on the order of 0.5 to 2 points, and the ablations involve choices of N, k, and T, reporting mean and standard deviation over at least three seeds, or providing per-seed numbers, is necessary to establish that the differences are not optimization noise. This is particularly important because the Charades-STA configuration was selected on the test set.","section":"§4.1 and §4.2"}],"minor_comments":[{"comment":"The abstract says the method outperforms 11 state-of-the-art methods, while the conclusion says ten leading methods; please reconcile the count.","section":"Abstract and Conclusion"},{"comment":"The text says 'As illustrated in Figure 4' when discussing the effect of k on performance, but the relevant figure appears to be Figure 6 (hyperparameter optimization results).","section":"§4.3, Fig. 6"},{"comment":"Equation (2) sets \\hat d_0 = max(d), which forces the first frame to be a key frame even though there is no preceding frame; this choice is not motivated and deserves a one-sentence justification.","section":"§3.3, Eq. (2)"},{"comment":"The LoRA rank and alpha are not reported; including them would improve reproducibility.","section":"§4.1"},{"comment":"For Charades-STA, mAP@0.5 and mAP@0.75 are reported only for LLaVA-MR and not for Mr. BLIP; adding these numbers for the closest baseline would help the reader assess the comparison.","section":"Table 1"},{"comment":"Variance-Based DTC is described as retaining the top half of queries by variance, but Table 7 later tunes T (the compressed token count) as a hyperparameter; please clarify whether the 'top half' rule is a fixed design choice or a tuned hyperparameter.","section":"§3.4 and Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable engineering contribution, but the evidence for the IFS mechanism is currently weak and the test-set selection on Charades-STA is a fairness concern. I would encourage the editor to request the additional ablations (IFS-only, DTC-only, random key-frame control) and a validation-based hyperparameter selection procedure. If the authors deliver those and resolve the Table 1/Table 3 inconsistency, the paper could become acceptable. The related-work coverage is adequate and I see no novelty-disclosure concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a credible engineering contribution with a real hole in the ablation logic. The idea is to take Mr. BLIP, sample frames densely, add time encoding, pick key frames by adjacent-feature distance, and compress non-key frames. Each piece is known; the combination for generative moment retrieval is new, and the QVHighlights test-set gains are credible because those numbers come from a server.\n\nWhat the paper does well: the system beats Mr. BLIP on all three benchmarks, the ablations are monotone, and the ActivityNet Captions numbers in the supplement are consistent. The motivation for dense sampling in long videos is sensible, and the inference time measurement is a nice touch.\n\nThe soft spots are substantial. The biggest one: IFS is never tested on its own. Table 2 adds IFS and DTC together, so the +2.13 gain belongs to the pair. Table 5 varies only the compression method with IFS fixed, so the selection criterion itself is never tested. A random key-frame control would settle this, but it's absent. Second, the ablation baseline undercuts the story: Table 2's sparse baseline is 65.48 on Charades-STA, while Table 1 quotes Mr. BLIP at 69.31 on the same split, so the marginal gains are measured from a weaker base and look inflated. Third, hyperparameters N, k, and T are selected on the test set for Charades-STA, and there are no error bars or multiple seeds. Fourth, the time encoding section has a direct contradiction: the prose says frame indices win on QVHighlights, but Table 3 shows timestamps win by 1.29. The post-processing heuristics are also described vaguely; their effect on the final numbers is unknown.\n\nNone of this is fatal to the overall claim that dense sampling plus smart compression helps long-video MLLMs. But the central mechanism, IFS, is not established as load-bearing. The paper would benefit from a proper referee: the IFS isolation, the baseline alignment, and the time encoding contradiction need to be addressed. I'd send it to review, but I'd want those points fixed before treating it as citable.","headline":"Plausible engineering recipe let down by an un-isolated key module and some sloppy evaluation; worth review, but fix the ablations first.","tokens_in":16779,"tokens_out":4777,"would_cite":false,"duration_ms":41690,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By sampling densely, flagging the most dynamic frames, and compressing the rest, LLaVA-MR outperforms prior state-of-the-art methods on QVHighlights, Charades-STA, and ActivityNet Captions.","keywords":["video moment retrieval","multimodal large language model","dense frame sampling","time encoding","informative frame selection","dynamic token compression","QVHighlights","Charades-STA"],"falsifier":"Run a controlled ablation that keeps the total token budget identical but replaces Informative Frame Selection's top-distance key frames with the same number of uniformly random key frames, repeated over several seeds on Charades-STA and QVHighlights; if the R1@0.5 and mAP differences stay within noise, the adjacent-feature-distance criterion is not doing the claimed work.","tokens_in":15762,"feed_emoji":"🎬","tokens_out":6603,"duration_ms":62781,"temperature":0.7,"pith_summary":"LLaVA-MR is a recipe for making a generative multimodal large language model do video moment retrieval on long videos, where sparse frame sampling and limited LLM context normally force a trade-off between coverage and precision. The paper claims that three coordinated changes—dense sampling with a per-frame time token, selection of the most dynamic frames, and compression of the remaining frame tokens—let the model locate query-relevant intervals better than the leading MLLM baseline and prior transformer-based methods. On QVHighlights test it reports 76.59 R1@0.5 and 69.41 mAP@0.5, on Charades-STA 70.65 R1@0.5, and on ActivityNet Captions 55.16 R1@0.5, each above Mr. BLIP. If the claim holds, it suggests the bottleneck for MLLM-based moment retrieval is not the language model's understanding but how frames are selected and tokenized, and that task-specific prediction heads are not needed.","feed_headline":"Dense frames and key-frame selection top moment retrieval","feed_subtitle":"A multimodal LLM assistant beats prior methods on three benchmarks by keeping only the most dynamic frames.","key_machinery":"The load-bearing machinery is the interleaved multimodal token sequence fed to the LLM, built by three modules. DFTE increases the number of sampled frames (60 for Charades-STA, 80 for QVHighlights) and places a time token before every frame embedding, using relative frame indices when sampled frames per second is at least one and timestamps otherwise. IFS computes the L2 distance between adjacent frame feature maps, smooths the resulting signal with a Gaussian filter, and treats the top-k frames as key frames that retain their full Q-Former tokens. DTC compresses the remaining non-key frames, either by average pooling or by keeping only the half of Q-Former queries with the highest variance across frames, which the paper argues preserves dynamic content. The sequence, marked with begin and end tokens for time and frame segments, is fed to the LLM, which outputs a nested list of predicted start and end times.","core_discovery":"The central claim is that moment retrieval can be reframed as an open-ended text-generation problem solved by a pretrained MLLM, provided the video is represented as a dense but compressed interleaved sequence of time tokens and frame embeddings. Dense Frame and Time Encoding supplies fine-grained spatial-temporal features by sampling many frames and prefixing each with a time token chosen by the frame sampling rate. Informative Frame Selection marks the frames with the largest Gaussian-smoothed adjacent-feature L2 distances as key frames, capturing event boundaries, and Dynamic Token Compression reduces the token count of non-key frames by keeping only Q-Former queries with the highest variance across frames. The paper reports that this combination reaches state-of-the-art results on QVHighlights, Charades-STA, and ActivityNet Captions, with the largest gains on the longer-video dataset.","pith_inferences":["If the adjacent-feature-distance criterion is truly what selects key frames, the same selection module should transfer to other long-video MLLM tasks with event boundaries, such as temporal action localization or highlight detection, without retraining the selector.","A possible extension the paper does not develop is query-aware key-frame selection: IFS is query-agnostic, so reranking candidate key frames by relevance to the text could improve recall on moments that are visually static but semantically relevant.","The sampling-rate rule for time encoding suggests a continuous time embedding instead of integer indices or rounded timestamps, which should remove the token-collision problem and may improve both dense and sparse sampling regimes."],"forward_implications":["MLLM-based moment retrieval can use far more frames than the LLM context would naively allow, because most frames are reduced to a few high-variance query tokens.","The choice of time representation matters: relative frame indices beat rounded timestamps when frames are denser than one per second, and timestamps beat indices when frames are sparser.","Variance-based token compression outperforms both no compression and average pooling, indicating that high-variance Q-Former queries carry the dynamic information the task needs.","The full pipeline runs faster than the sparse-sampling baseline Mr. BLIP at the same frame count, because compression more than offsets the cost of frame selection.","The method transfers to longer and more complex videos, with larger absolute gains on QVHighlights than on Charades-STA."],"supporting_citations":[{"why":"Establishes the generative MLLM baseline and training/inference protocol that LLaVA-MR extends; the main comparison to beat.","marker":"[28]"},{"why":"Supplies the frozen BLIP-2 image encoder, Q-Former, and LLM backbone whose frame tokens are selected and compressed.","marker":"[17]"},{"why":"Provides the QVHighlights benchmark with 150-second videos and hidden test evaluation used for the headline results.","marker":"[15]"},{"why":"Provides the Charades-STA benchmark and the TALL baseline for moment retrieval on short daily-activity videos.","marker":"[7]"},{"why":"Supplies the LoRA parameter-efficient fine-tuning that keeps training cost at 0.63 percent of total parameters.","marker":"[9]"},{"why":"Provides the ActivityNet Captions benchmark used in the supplementary comparison on two-minute videos.","marker":"[12]"}],"fun_headline_variants":["LLaVA-MR turns moment retrieval into open-ended text generation","Dense frame encoding and token compression sharpen moment search","LLaVA-MR beats 11 SOTA methods on video moment retrieval","Key-frame selection and token compression locate video moments","Multimodal LLM locates moments in long videos with dense frames"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that the frames whose adjacent visual features change most sharply are the frames that matter for locating the queried moment, and that the remaining frames can be compressed without losing information the model needs.","fun_headline_variants_meta":{"raw":{"variants":["LLaVA-MR turns moment retrieval into open-ended text generation","Dense frame encoding and token compression sharpen moment search","LLaVA-MR beats 11 SOTA methods on video moment retrieval","Key-frame selection and token compression locate video moments","Multimodal LLM locates moments in long videos with dense frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001031,"raw_usage":{"total_tokens":4318,"prompt_tokens":897,"completion_tokens":3421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":3334}},"tokens_in":513,"tokens_out":3421,"duration_ms":22597,"temperature":1.0,"reasoning_tokens":3334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:40:33.863523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled ablation that keeps the total token budget identical but replaces Informative Frame Selection's top-distance key frames with the same number of uniformly random key frames, repeated over several seeds on Charades-STA and QVHighlights; if the R1@0.5 and mAP differences stay within noise, the adjacent-feature-distance criterion is not doing the claimed work.","supporting_citations":[{"cited_title":"The surprising effectiveness of multimodal large language models for video moment retrieval, 2024","cited_arxiv_id":null,"evidence_quote":"Establishes the generative MLLM baseline and training/inference protocol that LLaVA-MR extends; the main comparison to beat."},{"cited_title":"Detecting mo- ments and highlights in videos via natural language queries","cited_arxiv_id":null,"evidence_quote":"Provides the QVHighlights benchmark with 150-second videos and hidden test evaluation used for the headline results."},{"cited_title":"Tall: Temporal activity localization via language query","cited_arxiv_id":null,"evidence_quote":"Provides the Charades-STA benchmark and the TALL baseline for moment retrieval on short daily-activity videos."}],"review_version":1}