{"id":"d1273576-a777-4157-88ad-185ea10afc21","arxiv_id":"2411.16173","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SALOVA improves long-form video understanding by routing question-relevant video segments into the language model, outperforming prior video-LLMs on Video-MME and LongVideoBench.","lead":"A new video-LLM framework, SALOVA, retrieves only the most relevant segments from long videos before answering questions, rather than processing every frame. It is built around the 87.8K-video SceneWalk dataset with dense segment captions and a segment-routing mechanism that improves accuracy on long-form video benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No controlled ablation isolates the Segment Retrieval Router: Table 3's 1 FPS 'w/o SR-Router' row matches the full pipeline (45.3 overall), so the claim that retrieval/routing drives the gains is not demonstrated.","rationale":"Good-faith reading: SALOVA proposes a coherent retrieval-augmented pipeline, a large densely-captioned dataset, and careful training. The strongest evidence for the retrieval claim would be an ablation in which the router is the only variable. Table 3 does not provide this: its first block is labeled 'without SR-Router,' and the 1 FPS row in that block is numerically identical to the full pipeline in every duration bucket. Either the table contains a repeated row, in which case there is no no-router baseline at 1 FPS, or the router truly adds nothing at 1 FPS. Both readings undermine the central claim. The reader's threshold-based concern is real but secondary: mis-specified supervision would degrade the router, yet the data here suggest that even a well-specified router may not be what moves the benchmark numbers. The Stage 1.5 and FocusFast ablations show other parts of the recipe help, so the paper is not hollow; but the 'targeted retrieval and routing' contribution, which is the paper's stated novelty, lacks a decisive controlled test. The comparison to LongVA is additionally confounded by different LLM backbones, training data, and token budgets, so the 0.5-point Video-MME gap cannot be attributed to retrieval either. A conditional verdict is appropriate: the method is plausible and partially supported, but the central causal claim is unverified. No code or dataset is released, so independent reproduction is currently impossible; the proposed seed-based controlled ablation would settle the attribution question.","tokens_in":21520,"tokens_out":6352,"duration_ms":74440,"concrete_test":"Run a controlled ablation on SALOVA-3B with identical Stage 1.5 and Stage 2 data and hyperparameters: (A) full pipeline with SR-Router and top-K=9; (B) SR-Router removed and all 1 FPS segments fed directly to the LLM; (C) as (B) but with a matched token budget, e.g., uniformly sampled 9 segments or randomly selected K segments; (D) 8/16-frame sampling as in Table 3. Report Video-MME Short/Mid/Long/Overall with at least 3 seeds and standard errors. If (A) does not beat (B) or (C) beyond seed noise, the central claim that retrieval and routing improve long-video understanding is unsupported, and the paper's contribution should be reframed around dense sampling plus SceneWalk knowledge injection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that targeted segment retrieval and routing, not merely denser sampling, improves long-video understanding. The paper's own ablations do not isolate this component. Table 3's first block is explicitly labeled 'Frame sampling rate (w/o SR-Router)': 8 frames gives 42.5 overall on Video-MME, 16 frames gives 43.6, and dense 1 FPS sampling gives 45.3. The full SALOVA-3B pipeline with the SR-Router and top-K retrieval gives exactly 48.3/46.3/41.1/45.3 on Short/Mid/Long/Overall, the same overall and per-duration numbers as the 1 FPS no-router row. So at fixed 1 FPS sampling, removing the router changes nothing on this benchmark; the visible gains in this block come from increasing frame density. The Stage 1.5 ablation (+1.7 overall) and FocusFast ablation (+8.4 overall) are also not router ablations. The V-NIAH comparison in Fig. 3 compares 16-frame sampling against 1 FPS sampling and therefore conflates retrieval with sampling density. Given that the headline comparison against LongVA (52.6 vs 53.1) is not controlled for backbone, training data, or compute, the only internal evidence that 'segment retrieval and routing' is the active ingredient is missing. Even if the LanguageBind and SBERT thresholds in Section A.2 are perfect, the router could be well-trained and still contribute nothing here.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SALOVA, a video-LLM framework for long-form video understanding based on segment-level retrieval and routing. The authors introduce the SceneWalk dataset, comprising 87.8K long untrimmed YouTube videos partitioned into 1.29M segments with dense captions generated by VILA-1.5-13B. The architecture consists of a CLIP/SigLIP vision encoder, a Perceiver-style spatio-temporal connector, a two-layer Segment Retrieval Router trained with a similarity loss, and a FocusFast mechanism that combines top-K retrieved segment features with global routing tokens. The model is trained in three stages: cross-modal alignment, SceneWalk-based knowledge injection, and video instruction tuning. Experiments on Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, and MVBench report competitive numbers, including 53.1 overall on Video-MME without subtitles for SALOVA-7B versus 52.6 for LongVA-7B. The paper's central claim is that targeted segment retrieval and routing, rather than only denser sampling, improves long-video understanding.","tokens_in":21864,"tokens_out":6700,"duration_ms":62782,"significance":"If the central claim were established, the work would be a meaningful step: the SceneWalk dataset is large and segment-level dense captions of long untrimmed videos are genuinely useful for long-video research, and the retrieval-routing design is a clean way to bypass context-length limits. The paper also gives an explicit retrieval supervision objective and evaluates on multiple established benchmarks, which is a strength. However, the headline result is not currently supported by the internal ablations: the only frame-sampling ablation in Table 3 appears to show that dense 1 FPS sampling without the SR-Router matches the full pipeline exactly, so the paper does not isolate the contribution of retrieval and routing. No code or trained models are released in the manuscript, and all benchmark numbers are single runs without error bars or significance estimates. The contribution is therefore promising but conditional on a controlled ablation of the router and on reconciling the table interpretation.","major_comments":[{"comment":"The first block of Table 3 is labeled 'Frame sampling rate (w/o SR-Router)', but its '1 FPS' row reports exactly the same Video-MME numbers (48.3/46.3/41.1/45.3) as the full SALOVA-3B configuration in the same table. As printed, this means that at fixed 1 FPS sampling, removing the Segment Retrieval Router changes nothing on Video-MME, so the observed gains in this block could come entirely from denser frame sampling. This is the missing controlled ablation for the paper's central claim. Please either add a true controlled row (full pipeline versus the same pipeline with the SR-Router removed, at fixed 1 FPS and matched token budget) or correct the table labeling if the '1 FPS' row is intended to represent the full model. The Top-K ablation in Appendix D does not fill this gap because it varies the number of retrieved segments without comparing against a no-router baseline.","section":"§5.3, Table 3 (first block)"},{"comment":"The V-NIAH comparison in Fig. 3 compares 'SALOVA-3B (16 frm sample)' against 'SALOVA-3B (1 FPS sample)', so the frame count and the presence of the retrieval/routing mechanism are varied simultaneously. This does not demonstrate that retrieval improves needle localization; the improvement could be due to the much larger number of input frames. A controlled comparison at equal frame count or equal token budget is needed before the figure can support the routing claim.","section":"§5.3, Fig. 3"},{"comment":"The headline comparison of SALOVA-7B (53.1) against LongVA-7B (52.6) on Video-MME is not controlled for backbone model, training data, vision encoder, or compute, and all benchmark numbers are reported as single runs with no error bars or statistical significance. This is not by itself disqualifying for a systems paper, but the abstract and Section 5.2 present the improvement as evidence for the retrieval mechanism. Please soften the claim or add controlled comparisons that at least match backbone and training data, and report variances or multiple seeds where feasible.","section":"§5.2, Table 1"},{"comment":"The retrieval supervision signal y_i used to train the SR-Router is derived by binarizing LanguageBind and SBERT similarity matrices with fixed thresholds tau_V2T=0.18 and tau_T2T=0.8 and taking the union. No sensitivity analysis, ablation, or qualitative validation of these thresholds is provided. Since the router learns entirely from this proxy supervision, the central retrieval claim depends on the thresholds being reasonable. Please include a threshold sensitivity study or at least qualitative retrieval examples that verify the positive segments are genuinely relevant to the queries.","section":"§A.2 and Eq. (1)"}],"minor_comments":[{"comment":"The rows labeled '! ✗ !' and '! ! !' for the training-stage ablation are not defined in the table caption or in the main text; please spell out which stages are included and excluded.","section":"§5.3, Table 3"},{"comment":"The paper uses inconsistent spelling of the model name, alternating between 'SALOVA' (abstract) and 'SALOV A' (body and figures); please choose one and use it consistently.","section":"Throughout"},{"comment":"Section 5.1 says the 7B 'frontier model' uses SigLIP, but Appendix D reports LongVideoBench results for 'SALOVA-7B†' using CLIP at smaller resolution; please clarify explicitly which configuration produced the Table 1 headline numbers.","section":"§5.1 vs. Table 6"},{"comment":"The notation 'L_bce(y_i, s_i)_{N_v}_{i=1}' is malformed; it should be written as a sum over i, e.g., (1/N_v) sum_i BCE(y_i, s_i), to be unambiguous.","section":"Eq. (1)"},{"comment":"The sentence 'Among the total32 coarse and diverse video categories YouTube API provided' has a missing space and should read 'total 32'; please also fix the duplicated '(ii)' in the list of video source criteria.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The central claim is plausible but not yet demonstrated by the manuscript's own ablations. The most important fix is a controlled SR-Router ablation at fixed sampling density; without it, the paper cannot support 'targeted retrieval and routing' as the active ingredient. I would also suggest verifying the LongVideoBench numbers across Table 1 and Table 6, since the reader may otherwise doubt reproducibility. The SceneWalk dataset is a valuable contribution, but the manuscript does not state whether the dataset or model checkpoints will be released; clarifying this would help the community use the work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a serious long-video QA paper with a genuinely useful new dataset and a plausible retrieval-augmented architecture, but its central claim—that the segment router drives the gains—is not supported by the paper's own ablations. On Video-MME, the full 3B model with the router and top-K retrieval scores 45.3 overall; the '1 FPS w/o SR-Router' row in Table 3 scores exactly 45.3 with identical per-duration breakdowns. So at a fixed 1 FPS sampling density, removing the router changes nothing. The visible improvements in that table come from increasing frame density, not from routing. The V-NIAH comparison in Fig. 3 similarly conflates retrieval with sampling density.\n\nWhat is actually good: the SceneWalk dataset is a substantial resource—87.8K long videos, 1.3M segments, dense captions averaging 137 words, with a thoughtful two-signal supervision scheme using LanguageBind and SBERT. The architecture is coherent: the spatio-temporal connector, segment retrieval router, and FocusFast dual-pathway design are standard components but put together in a new way. The reported numbers are competitive—53.1 on Video-MME without subtitles for the 7B variant edges out LongVA's 52.6, and 44.6 on LongVideoBench is decent. The qualitative examples show the model can localize relevant scenes in long videos.\n\nSoft spots, in order of severity. First, the ablation problem above is load-bearing. The paper concludes 'using more frames with SR-Router significantly enhances performance,' but that is not what the table shows; the router is orthogonal to the frame-count axis. Second, the comparison against LongVA is not controlled: different backbone, different training data, different compute, no error bars or multiple seeds. The 0.5-point overall margin could easily be noise. Third, the retrieval supervision thresholds and top-K are manually chosen, and the top-K ablation shows a peak at 9 while the main pipeline uses 5 during training—a free parameter that deserves sensitivity analysis. Fourth, no code or dataset release limits verification.\n\nWho this is for: the long-video QA community, especially people working on RAG-style video-LLMs. The SceneWalk dataset, if released, could be a useful training resource. The paper deserves a serious referee—the architecture is worth discussing and the dataset is valuable—but the authors should be asked to provide a proper ablation that holds sampling density fixed while ablating the router, and ideally error bars and a controlled comparison. I would not desk-reject, but the central claim needs stronger evidence.","headline":"A serious long-video QA paper with a useful new dataset and plausible architecture, but its own ablations fail to isolate the router's contribution, so the central claim is not actually demonstrated.","tokens_in":22360,"tokens_out":2170,"would_cite":false,"duration_ms":20850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SALOVA claims that a video language model that retrieves only the segments relevant to a user's question and routes them into the model can understand hour-long untrimmed videos better than models that sample or compress the entire video…","keywords":["long video understanding","video large language model","retrieval-augmented generation","segment retrieval","video question answering","SceneWalk dataset","video instruction tuning","Video-MME"],"falsifier":"Build a subset of Video-MME where the segment containing the correct answer has low video-text and text-text similarity under the paper's thresholds, then run SALOVA-7B on it; if accuracy on this subset is no better than the 16-frame no-router baseline, the router's retrieval supervision, not the architecture, is responsible for the reported gains.","tokens_in":21354,"feed_emoji":"🎬","tokens_out":7801,"duration_ms":63320,"temperature":0.7,"pith_summary":"SALOVA claims that long-form video understanding improves when a video-LLM retrieves only the segments of an untrimmed video that are relevant to the user's question and routes those segments into the language model, instead of sampling or compressing the whole video into a fixed context. The paper contributes the SceneWalk dataset—87.8K long web videos split into 1.29M segments with dense descriptions (137.5 words each on average)—and a training pipeline that teaches the model which segments matter for a query, then fuses the retrieved details with a compact global representation. On Video-MME without subtitles, SALOVA-7B reaches 53.1 overall, ahead of LongVA's 52.6, with the largest gains on medium and long videos; it also scores 44.6 on LongVideoBench validation. If the claim holds, retrieval-and-routing offers a practical way to handle hour-long videos within today's context-length limits while keeping memory overhead low.","feed_headline":"Routing only relevant clips beats LongVA on long-video QA","feed_subtitle":"SALOVA-7B scores 53.1 on Video-MME without subtitles, surpassing LongVA's 52.6 with a retrieval-first design.","key_machinery":"The mechanism has three cooperating pieces. First, the SceneWalk dataset provides dense per-segment captions (137.5 words on average) from 87.8K long videos, giving the model training signal about scene continuity across segment boundaries. Second, the Segment Retrieval Router—a two-layer transformer that attends to query text embeddings—produces video-to-text similarity scores, and a margin-based similarity loss trains it to match the precomputed correspondence matrices; the object is to decide which segments to route. Third, the FocusFast pathway splits the LLM input into a focus stream (the top-5 retrieved segment features, processed densely) and a fast stream (routing tokens from all segments, providing global context), so the model can answer detailed questions without losing the surrounding narrative. A dynamic token-drop mechanism in the spatio-temporal connector keeps the computational cost of processing long inputs manageable.","core_discovery":"The central claim is that a retrieval-driven architecture can outperform whole-video processing for long, untrimmed videos. SALOVA segments the video, embeds each segment with a spatio-temporal connector, and uses a segment retrieval router to score how well each segment matches the user's query; it then feeds the top-scoring segments to the LLM while also passing compact routing tokens from all segments to preserve global context (the FocusFast pathway). The router is trained end-to-end with a similarity loss whose supervision comes from precomputed video-text and text-text correspondence scores, binarized with thresholds $\\tau_{\\mathrm{V2T}}=0.18$ and $\\tau_{\\mathrm{T2T}}=0.8$ and combined as a union. The paper reports that this design lets SALOVA-7B surpass prior open video-LLMs on Video-MME (53.1 overall, without subtitles) and set strong results on LongVideoBench (44.6 validation), with ablations showing the SceneWalk knowledge-injection stage and the FocusFast pathway are each responsible for part of the gain.","pith_inferences":["The paper leaves implicit that the fixed thresholds on precomputed similarity scores are the weakest link in the supervision; a version that learns the relevance threshold jointly, or filters segments with human-reviewed labels, would test how much of the gain is attributable to the retrieval supervision itself.","Because the reported benefit concentrates on videos longer than four minutes, one direct extension is a hybrid model that decides per video whether to use dense sampling or segment routing, which the paper itself floats as a future direction.","The same segment-and-route idea could transfer to other long sequential modalities such as audio or instructions, where a query-specific router could select the relevant portion of a long stream before expensive reasoning.","A testable extension would be to run SALOVA on streaming or incremental video inputs, where segments arrive over time and the router would need to decide when to update the retrieved set."],"forward_implications":["Long-video QA can be scaled to hour-long inputs by routing only the relevant segments into the LLM, reducing the token and memory footprint compared with dense frame sampling.","The SceneWalk knowledge-injection stage is a load-bearing part of training: skipping it drops Video-MME overall from 45.3 to 43.6 in the 3B model, and the gains are largest on medium and long videos.","The FocusFast fusion of retrieved details with global routing tokens contributes substantially to accuracy; removing it lowers the 3B model's overall score from 45.3 to 36.9.","The top-$K$ retrieval count matters: increasing from 1 to 9 segments improves Video-MME overall from 43.9 to 45.3, after which performance saturates, indicating the pipeline's sweet spot."],"supporting_citations":[{"why":"Serves as the main long-context baseline on Video-MME and the source of the V-NIAH evaluation used to show the router finds sparse evidence in long videos.","marker":"[70]"},{"why":"Supplies the video-to-text embeddings for the correspondence scores that define the router's retrieval supervision.","marker":"[73]"},{"why":"Supplies text-to-text similarity scoring used as the second, doubly-robust source of correspondence supervision.","marker":"[54]"},{"why":"Provides the Video-MME benchmark and its short/medium/long split, the primary evidence that SALOVA beats prior open video-LLMs.","marker":"[20]"},{"why":"Provides the LongVideoBench benchmark used to measure performance on videos up to hours long.","marker":"[57]"},{"why":"Provides the pre-trained captioning model that generates the dense segment captions in SceneWalk.","marker":"[36]"},{"why":"Supplies the SlowFast pathway concept on which the FocusFast local-global representation is built.","marker":"[19]"},{"why":"Provides the Perceiver Resampler architecture that the spatio-temporal connector adapts for fixed-size segment embeddings.","marker":"[2]"}],"fun_headline_variants":["Retrieval-first video LLM beats whole-video processing on long QA","SALOVA: pinpoints relevant clips to ace long-video comprehension","Segment routing tops LongVA on Video-MME long-video tests","SALOVA retrieves only needed clips, beats LongVA on long video","Targeted retrieval: SALOVA trumps LongVA on long-video QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training signal for which video segments matter comes from precomputed similarity scores that are turned into binary labels with fixed cutoffs; if those scores misjudge relevance, the router learns to retrieve the wrong segments and the reported benefit of the pipeline rests on that proxy.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-first video LLM beats whole-video processing on long QA","SALOVA: pinpoints relevant clips to ace long-video comprehension","Segment routing tops LongVA on Video-MME long-video tests","SALOVA retrieves only needed clips, beats LongVA on long video","Targeted retrieval: SALOVA trumps LongVA on long-video QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000875,"raw_usage":{"total_tokens":3818,"prompt_tokens":1009,"completion_tokens":2809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2721}},"tokens_in":625,"tokens_out":2809,"duration_ms":19381,"temperature":1.0,"reasoning_tokens":2721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:26:32.258801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a subset of Video-MME where the segment containing the correct answer has low video-text and text-text similarity under the paper's thresholds, then run SALOVA-7B on it; if accuracy on this subset is no better than the 16-frame no-router baseline, the router's retrieval supervision, not the architecture, is responsible for the reported gains.","supporting_citations":[{"cited_title":"Provide a detailed description of both the visual content and the storyline depicted in the video","cited_arxiv_id":null,"evidence_quote":"Supplies the video-to-text embeddings for the correspondence scores that define the router's retrieval supervision."},{"cited_title":"Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks","cited_arxiv_id":null,"evidence_quote":"Supplies text-to-text similarity scoring used as the second, doubly-robust source of correspondence supervision."},{"cited_title":"Vila: On pre-training for vi- sual language models","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained captioning model that generates the dense segment captions in SceneWalk."},{"cited_title":"Slowfast networks for video recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the SlowFast pathway concept on which the FocusFast local-global representation is built."}],"review_version":1}