{"id":"90dfe78e-80b8-4077-8d0e-8c4b3277ade1","arxiv_id":"2412.09276","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new task, dataset, and multi-grained text-video fusion model for automatically assembling video montages from a narration script.","lead":"This paper introduces Video Moment Montage, a task where a system automatically cuts and assembles short video clips to match a multi-sentence narration script. It also presents a new large-scale dataset and a multi-grained text-video fusion method that outperforms several adapted baselines on that dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set may preserve original shot order, letting TV-MGI exploit positional alignment with the script rather than semantic match; a shuffled re-evaluation would settle whether the reported gains are a shortcut.","rationale":"The reader's weakest assumption was test set representativeness. I identify a specific mechanism: the test set may preserve shot order, giving the model a positional shortcut. This is more load-bearing because it directly threatens the internal validity of the comparison: the model's positional encoding plus ordered shots could solve the task without fine-grained semantic matching. The paper's training uses Shuffling to simulate unordered materials, yet the test set construction is ambiguous on this point, and the strong quantitative gains make the shortcut plausible. This is not an ad hominem or consensus disagreement; it is a checkable property of the test protocol. The user study exclusion of RATV and PREM and missing baseline tuning details are secondary concerns that further support conditionality, but the shot-order issue is the single most actionable one. If the shuffled test shows the method remains top, the concern is resolved; if not, the central claim requires qualification. The verdict remains CONDITIONAL (no change) because the paper already needs code/data release and more rigorous evaluation; this concern reinforces those conditions without demanding rejection.","tokens_in":12125,"tokens_out":7327,"duration_ms":70878,"concrete_test":"Inspect the released MSSD test set: determine whether candidate shots within each test instance are shuffled or kept in original temporal order. Then run the trained TV-MGI and the strongest baseline (e.g., PREM) on a shuffled version of the test set, where the order of all candidate shots is randomized while ground-truth segments remain fixed. If the method's R1@0.5 and mAP@5 drop by more than a few points or the margin over PREM narrows substantially, the reported superiority partially relies on temporal-order cues. Releasing the test set and this shuffled variant would allow independent verification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('outperforms all baseline methods across all metrics') is evaluated on the MSSD test split. The test set is constructed by clustering videos, pairing two per cluster, and designating one video's aligned shots as ground truth (Dataset Construction). If the candidate shots are placed in their original chronological order, then the ground-truth video's shots appear in the same order as the script sentences. TV-MGI adds positional encodings to both text and video features (Multi-grained Fusion Encoder), so the model can learn to prefer the video whose shot order correlates with sentence order instead of matching semantics. The paper's Synthetic Data Augmentation explicitly uses a Shuffling strategy for training to mimic unordered raw materials, but the test set description does not state whether shots are shuffled before being presented as candidate materials. If shuffling is absent, the test distribution is easier and unrepresentative of real editing workloads, where footage is unordered. The reported margins (e.g., R1@0.5 30.91 vs. 17.35) could therefore reflect an order-matching shortcut rather than robust text-video alignment. This is a concrete, testable confound rather than a general representativeness worry.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Video Moment Montage (VMM), a task in which a model must localize and assemble video shots from a set of raw footage videos to match a multi-sentence narration script. The authors propose TV-MGI, which fuses CLIP text and frame features with shot-level and frame-level cross-attention, adds a rank-aware contrastive loss, and uses synthetic data augmentation with shot shuffling and injection. They also build a new large-scale dataset, MSSD, with 187k videos and ASR/OCR-derived scripts and sentence-shot alignments. On a synthetic test set of 2,921 paired-video instances, they report that TV-MGI outperforms five baselines across all metrics, with ablations and a user study.","tokens_in":12387,"tokens_out":4968,"duration_ms":45541,"significance":"If the results are robust, the paper makes a solid contribution by proposing a practically motivated task and a large dataset. The MSSD dataset itself is a substantial resource (187k videos, 808k sentences). The TV-MGI model is a sensible combination of multi-grained fusion and ranking losses, and the ablations show each component contributes. However, the evaluation is vulnerable to a positional-order shortcut in the test construction and to under-specified baseline adaptation; both need to be addressed before the superiority claim is established.","major_comments":[{"comment":"The test set description in the Dataset Construction section does not state whether the candidate shots are shuffled before being presented as materials. Because the fusion encoder adds positional encodings to both text and video features, and because the training augmentation explicitly uses Shuffling to mimic unordered footage, the model could learn to rely on the chronological correspondence between the script sentence order and the shot order of the ground-truth video rather than semantic alignment. Since the test set is built by pairing two videos and using one video's aligned shots as ground truth, if the candidate list preserves the original shot order, the reported margins (e.g., R1@0.5 30.91 vs. PREM 17.35) may be inflated. Please clarify the test-set construction and, if shuffling is not applied, re-evaluate on a shuffled candidate set to verify that gains persist.","section":"Dataset Construction and Multi-grained Fusion Encoder"},{"comment":"The description of baseline adaptation is too brief to establish a fair comparison. It is unclear whether Moment-DETR, QD-DETR, RATV, and PREM were trained on the MSSD training set with the same optimization budget, or were used with default or pretrained weights and only inference. For a fair claim of 'outperforms all baseline methods across all metrics' (Table 1), the paper should specify fine-tuning procedures, hyperparameter selection, and any adaptation efforts (e.g., how RATV's video-level retrieval was converted to the sentence-shot setting). Without this, the baseline results may be undertuned, which would bias the comparison.","section":"Experimental Settings: Baselines"},{"comment":"The user study excludes RATV and PREM because their outputs 'exhibit a chaotic nature.' This selection means the user study does not compare against all six methods, and the conclusion about perceptual superiority is limited to the four retained methods. Please either include all methods in the study or explicitly phrase the claim as applying to the four-method comparison. Additionally, the study uses 13 volunteers with 10 sets each and only two rankings per set; the small sample should be acknowledged when claiming significance.","section":"User Study"},{"comment":"No error bars or significance tests are reported. Given that the test set has 2,921 instances, repeating the evaluation with different seeds or reporting confidence intervals would strengthen the claim that the margins are not due to noise, particularly for smaller gaps such as QD-DETR vs. Moment-DETR.","section":"Quantitative Analysis and Table 1"}],"minor_comments":[{"comment":"The text mentions trainable projection weights 'W t_m and W c_m', but the equation uses 'W v_m'; please correct the notation.","section":"Prediction and Loss (Eq. 4)"},{"comment":"The caption contains the typo 'evlautaion'; it should read 'evaluation'.","section":"Table 1"},{"comment":"The notation 'LvLf' for the total number of frames is ambiguous; write it as L_v·L_f or N_frames_total.","section":"Multi-grained Fusion Encoder"},{"comment":"The statement that 'A higher rank indicates an earlier appearance of the shot in time' is counterintuitive and conflicts with the usual understanding of rank; please clarify with an example, since the rank-aware loss depends on this definition.","section":"Method: Formal Definition"},{"comment":"The notation such as '(Videos_5; Start1_1, End 1_1)' in the figure is hard to parse; consider simplifying to make the example clearer.","section":"Figure 1"},{"comment":"The paper does not mention whether code or data will be released; an availability statement would aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper shares a co-author (Bo Wang) with the PREM baseline (Chen et al. 2024), which is not a conflict per se but heightens the need for the baseline adaptation description called for in the major comments. The MSSD dataset is a significant resource, but the evaluation confound around shot order is a correct-ability risk that should be resolved before publication. The paper may also benefit from a clearer statement of its scope relative to the journal's readership."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Yo,\n\nThe paper is worth a look: it defines a new task (VMM) and a large dataset (MSSD), and the method is a sensible combination of CLIP features, cross-attention fusion, and a rank-aware contrastive loss. The results on their test split are consistently better than the baselines, and the ablations show each component earns its keep. Good work on the data collection, too — 187k videos with ASR/OCR-aligned scripts is a tangible resource.\n\nThe soft spots are mostly in the evaluation. The biggest one is the test set construction. The paper says they use shots from two videos and 'perform synthetic data augmentation as candidate materials' to make each test instance, but it never says whether the shots are shuffled before being presented to the model. If the candidate shots stay in original video order, TV-MGI can exploit the positional encodings — matching sentence order to shot order — instead of doing semantic text-video alignment. That could explain the large margins (R1@0.5 30.91 vs 17.35 for the best baseline). The training-time shuffling strategy shows the authors know raw footage is unordered; the test set should reflect that or it's an unfair evaluation. This is testable — re-run on a shuffled test set — and a referee should ask for it.\n\nOther issues: no error bars or significance tests, so we don't know if the margins are stable; the user study drops RATV and PREM because their outputs are 'chaotic,' which biases the comparison; and baseline tuning is described in two sentences, making it hard to know if the baselines got a fair chance. None of these are fatal on their own, but together they make the headline claim 'outperforms all baselines across all metrics' fragile.\n\nThe citation pattern looks fine; the method is explicitly built on prior work, and the dataset is new. No code or data appears to be released, which is a problem for a dataset paper.\n\nBottom line: the task and dataset are solid contributions, and the method is plausible. The evaluation needs a shuffled test set, error bars, and a complete user study before I'd trust the performance claims. I'd send this to peer review — a good referee can tighten it. It's the kind of paper that will be cited for MSSD even if TV-MGI itself doesn't survive close scrutiny.\n\nRecommended action: engage, but require the test-set re-evaluation.","headline":"New task and dataset are real contributions, but the test set may let the model cheat with positional order; worth reviewing with that question asked.","tokens_in":12893,"tokens_out":2986,"would_cite":true,"duration_ms":29678,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Video Moment Montage and claims its TV-MGI model, which fuses script text with shot- and frame-level video features, localizes and assembles segments better than all tested baselines on the new MSSD benchmark.","keywords":["Video Moment Montage","text-video retrieval","multi-grained fusion","shot-level ranking","moment retrieval","video montage generation","temporal grounding","rank-aware contrastive loss"],"falsifier":"Take a set of videos actually assembled by human editors, with raw footage that contains the source shots plus unrelated distractors; if TV-MGI fails to recover the editors' segment choices at comparable recall to the MSSD numbers, the central claim of superiority for real montage workloads would be contradicted. More directly, rerunning the main comparison with ground-truth segments drawn from true editor timestamps instead of the clustered two-video proxy would settle it.","tokens_in":11921,"feed_emoji":"🎬","tokens_out":7123,"duration_ms":63939,"temperature":0.7,"pith_summary":"The paper establishes Video Moment Montage (VMM) as a task: given a multi-sentence narration script and several raw videos, a model must locate the right moments for each sentence and assemble them into one coherent video. It then claims that its TV-MGI model, which fuses script text with both shot-level and frame-level video features and trains with a rank-aware loss, solves this task better than adapted retrieval and moment-retrieval baselines. The authors also build MSSD, a large script-video dataset with 187,351 videos and 808,936 sentences, so VMM can be studied quantitatively. If right, this is a step toward automating the slow parts of short-video editing from an ordinary script.","feed_headline":"TV-MGI assembles narration-script montages, beating all baselines","feed_subtitle":"It fuses shot- and frame-level video features with script text, then ranks and trims shots to match each sentence.","key_machinery":"The load-bearing object is the multi-grained fusion encoder. In each of its layers, frame features are mean-pooled into shot features; then three X-MHA cross-modal attention steps fuse shot-text, frame-shot, and frame-text representations; the outputs feed a prediction head that scores each shot for each sentence, applies a rank-aware contrastive loss for shot ordering, and predicts frame-level start and end positions within matched shots via softmax. This is what carries the claim that global and fine-grained alignment can be learned jointly.","core_discovery":"On the paper's own terms, the discovery is that multi-grained text-video fusion works for montage: per encoder layer, shot-level pooling and three cross-modal attention steps (shot-text, frame-shot, frame-text) let a model both choose which shots belong to each sentence and decide where within those shots the segment begins and ends. With a rank-aware contrastive loss ordering matched shots, TV-MGI reports R1@0.5 of 30.91 versus 21.65 for the strongest baseline and higher mAP and NDCG at both sentence and script levels; ablations show each component contributes. The paper also claims that synthetic augmentation, which shuffles shots and injects shots from related videos, is what lets the model handle redundant raw footage.","pith_inferences":["If the approach generalizes beyond the synthetic test setup, it could be the core of an assistive editing tool that drafts a rough cut from a script, with a human refining the result.","The rank-aware loss on shot order might transfer to other composition tasks where output order matters, such as slideshow generation or story visualization.","Because the test set is built from clusters of two videos, the benchmark rewards discrimination between related footage; real editing often draws from many more sources, so gains may differ with larger material pools.","A natural next test is cross-domain evaluation with scripts and footage from a different platform or language to see whether the learned alignment and ranking transfer."],"forward_implications":["An editor could supply only raw footage and a narration script; the model returns a montage with each sentence matched to one or more trimmed video segments.","Multi-sentence context is necessary, not optional: treating script sentences independently drops performance substantially, so montage systems should model inter-sentence relations.","Shot-level representations plus a ranking objective give both recall and correct ordering, which standard moment retrieval lacks.","Synthetic training data with shuffled and injected shots can bridge the gap between aligned training clips and messy real footage.","Existing VMG, VMR, and VCMR methods are insufficient for script-level montage, which is why a purpose-built dataset is needed."],"supporting_citations":[{"why":"Provides the CLIP visual and text encoders whose aligned features are the base for all video-text matching.","marker":"Radford et al. 2021"},{"why":"Supplies ChineseCLIP, the specific pretrained encoder used to extract frame and sentence features.","marker":"Yang et al. 2022"},{"why":"Defines the X-MHA cross-modal attention module that performs shot-text, frame-shot, and frame-text fusion.","marker":"Li et al. 2022"},{"why":"Provides the rank-aware contrastive loss that orders matched shots within each sentence.","marker":"Hoffmann et al. 2022"},{"why":"TransNet v2 detects shot boundaries used to define the shots and temporal alignment.","marker":"Souček and Lokoč 2020"},{"why":"Moment-DETR is a main baseline and the strongest retrieval competitor the method is compared against.","marker":"Lei, Berg, and Bansal 2021"},{"why":"QD-DETR serves as a query-dependent moment-retrieval baseline.","marker":"Moon et al. 2023"},{"why":"RATV is the multi-sentence video montage generation baseline that the new task extends.","marker":"Yang et al. 2023"},{"why":"PREM is the video corpus moment retrieval baseline used for multi-pass sentence-level retrieval.","marker":"Hou et al. 2024"}],"fun_headline_variants":["Multi-grained text-video fusion beats baselines in montage","TV-MGI: shot-frame alignment for scripted video editing","New method links sentences to video shots for montage","MSSD: dataset for training video moment montage models","Rank-aware contrastive loss improves shot selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the synthetic test set, built by pairing videos and using one video's own aligned shots as ground truth, faithfully represents how editors really montage raw footage; if that proxy is too easy or artificial, the reported gains may not carry over.","fun_headline_variants_meta":{"raw":{"variants":["Multi-grained text-video fusion beats baselines in montage","TV-MGI: shot-frame alignment for scripted video editing","New method links sentences to video shots for montage","MSSD: dataset for training video moment montage models","Rank-aware contrastive loss improves shot selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1832,"prompt_tokens":927,"completion_tokens":905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":543,"tokens_out":905,"duration_ms":8005,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:31.362231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of videos actually assembled by human editors, with raw footage that contains the source shots plus unrelated distractors; if TV-MGI fails to recover the editors' segment choices at comparable recall to the MSSD numbers, the central claim of superiority for real montage workloads would be contradicted. More directly, rerunning the main comparison with ground-truth segments drawn from true editor timestamps instead of the clustered two-video proxy would settle it.","supporting_citations":[{"cited_title":"T.; Behrmann, N.; Gall, J.; Brox, T.; and Noroozi, M","cited_arxiv_id":null,"evidence_quote":"Provides the rank-aware contrastive loss that orders matched shots within each sentence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RATV is the multi-sentence video montage generation baseline that the new task extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PREM is the video corpus moment retrieval baseline used for multi-pass sentence-level retrieval."}],"review_version":1}