{"id":"e1ac0a21-dcd7-4a70-9752-67d3bd99bd02","arxiv_id":"2501.07554","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SST-EM is a weighted combination of VLM caption similarity, object-detection confidence, and ViT frame-similarity scores, fitted to human ratings to evaluate text-based video edits.","lead":"SST-EM is a new automated score for judging video edits, using a vision-language model, object detection, and a temporal smoothness check. The paper claims it tracks human quality ratings better than existing CLIP-based metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.962 correlation in Table 3 is not shown to be out-of-sample: weights are fit to human scores in §4.5 and the reported correlations are not tied to the held-out Validation Set from §3.1, so the central outperformance claim may be a fitting artifact.","rationale":"I agree with the reader's rejection, but I would place the primary load-bearing weight on the in-sample fitting/validation gap rather than on the reliability of human scores. Even if the human evaluations were perfectly measured, a regression fit to those scores on the same data does not validate the metric; the high correlation would be a fitting artifact. The reader's stated weakest assumption is human-rater reliability, though the rationale does also mention that correlations are not tied to a validation set and weights are unreported, so there is partial overlap. The temporal-sign contradiction is real but secondary because a fitted negative w3 could partially compensate; the missing weights and missing held-out evaluation are what make the central claim untestable as written. Since the authors provide a code repository, the proposed validation check is feasible and could settle the concern.","tokens_in":10913,"tokens_out":4930,"duration_ms":53116,"concrete_test":"Run the released GitHub code end-to-end: fit w1, w2, w3 with the loss in §4.5 on the optimization set only; freeze the weights; compute S_final on the separate Validation Set from §3.1; then compute Pearson, Spearman, and Kendall correlations against human scores on that validation set, and repeat for every baseline in Table 3. If SST-EM's validation correlations are not the highest, or if the split and weights cannot be reconstructed, then the §5.5 claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5.5, Table 3) is that SST-EM has the highest correlations with Human Evaluation scores, with Pearson 0.962. The load-bearing problem is that this number is not established as a held-out result. In §4.5–4.6, the final score S_final = w1·S_similarity + w2·S_object + w3·(1−S_temporal) has weights w1, w2, w3 fit by least squares to human scores on the optimization set, and the Pearson correlation in §4.6 is written over the same M videos used for training. Table 3 does not state that the correlations are computed on the Validation Set introduced in §3.1, and the fitted weights are never reported. With only 40 videos and 3 fitted weights, a high in-sample correlation is expected and does not demonstrate alignment with human judgment. The claim of 'highest correlations' therefore rests on a missing validation protocol. A secondary internal inconsistency appears in §4.4: §4.3 defines higher S_temporal as better temporal consistency, yet S_final subtracts S_temporal from 1; without the fitted weights one cannot tell whether the sign of w3 compensates, but as written the formula contradicts its own direction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SST-EM, a composite metric for evaluating text-based video editing, combining a PaliGemma-based semantic similarity score, a Grounding DINO object-detection score, and a ViT-based temporal consistency score. The three components are combined as a weighted sum, with weights fit by least-squares regression to human evaluation scores on a 40-video optimization set. The central claim is that SST-EM achieves the highest correlations with human evaluation scores among compared metrics, with a Pearson correlation of 0.962 (Table 3). The paper also reports ablation-style comparisons across several video editing models.","tokens_in":11181,"tokens_out":3002,"duration_ms":28758,"significance":"If the central claim were rigorously established, SST-EM would be a useful contribution to video-editing evaluation, where automated metrics that align with human judgment are genuinely needed. The component choices are reasonable and draw on modern tools (PaliGemma, Grounding DINO, ViT). However, the paper's headline evidence is currently in-sample and circular: the weights are fit to human scores and then correlated with those same scores, with no reported validation-set result. The temporal term in the final formula also has an apparent sign error. These issues make the current manuscript unsuitable for publication without substantial revision.","major_comments":[{"comment":"The headline result, Pearson ρ = 0.962 in Table 3, is not shown to be out-of-sample. Section 4.5 fits w1, w2, w3 by minimizing squared error against human scores on the optimization set, and Section 4.6 computes the Pearson correlation over the same M videos used for training. The Validation Set introduced in Section 3.1 is never tied to the correlations in Table 3, and the fitted weights are never reported. With only 40 videos and 3 free parameters, a high in-sample correlation is expected and does not substantiate the claim in Section 5.5 that SST-EM 'exhibits the highest correlations' with human judgment. The authors must report the validation-set correlation, the sample size for that set, and the fitted weights.","section":"§4.5–4.6, §5.1, Table 3"},{"comment":"The final-score formula S_final = w1·S_similarity + w2·S_object + w3·(1−S_temporal) is presented with the statement that subtracting S_temporal from 1 'ensures that a higher temporal consistency score yields a higher final score.' This is directionally wrong for positive w3: higher S_temporal makes (1−S_temporal) smaller, so the temporal term decreases. Unless the fitted w3 is negative, which is never stated or reported, the formula contradicts the paper's own description. This must be corrected and the sign of w3 disclosed.","section":"§4.4, Eq. (4)"},{"comment":"The human evaluation scores that serve as regression targets are described only as 'subjective ratings collected from multiple individuals.' The paper gives no rater count, annotation instructions, judgment scale, or inter-rater reliability, despite Section 2 explicitly noting that Krippendorff's alpha or Cohen's kappa are needed to ensure reliability. With only 40 videos in the optimization set, noisy or biased ground truth would propagate directly into the fitted weights and every subsequent correlation. This missing information is load-bearing for the validity of the entire evaluation.","section":"§3.1, §2"},{"comment":"The text states that SST-EM achieves Spearman and Kendall correlations of 1.000 'with both Imaging Quality and Aesthetic Quality,' but Table 3 reports those perfect correlations against Human Evaluation scores, not against Imaging Quality or Aesthetic Quality. This conflation of the reference variable obscures what is actually being compared and should be corrected.","section":"§5.3, §5.4"}],"minor_comments":[{"comment":"The notation 'w3 ·(S1−temporal)' appears to be a typo for 'w3 ·(1−S_temporal)'; please fix the formula formatting.","section":"§4.4"},{"comment":"The paper refers to an 'R1 score' without defining it; presumably this is R², and if so the symbol should be corrected and its computation described.","section":"§4.6"},{"comment":"The table reports 40 videos and 640 frame-prompt pairs for Weights-Optimization, but 40 videos and 900 pairs for Evaluation; the relationship between these counts and the optimization/validation split is unclear and should be clarified.","section":"§3, Table 2"},{"comment":"The Context Similarity Score has Pearson 0.072 and negative Spearman/Kendall correlations with human evaluation; the claim that 'individual components... also show notable correlation' is not supported for this component.","section":"§5.1, Table 3"},{"comment":"Some references appear misaligned with their cited claims, e.g., [22] is cited for regression-based metric optimization in video editing but the reference is about gaze-guided cinematic editing; please verify all citations.","section":"§1, §2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a promising pipeline but the evaluation protocol is currently circular. The authors should be asked to report held-out validation correlations, fitted weights, and full human-evaluation details. If the held-out results are strong, the paper could become acceptable; if not, the central claim collapses. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SST-EM is a plausible engineering contribution — a weighted combination of PaliGemma caption similarity, Grounding DINO object confidence, and ViT frame-embedding similarity, with the object prompt refined by an LLM agent — and the code is public. The problem is that the paper's central claim, a 0.962 Pearson and 1.0 Spearman/Kendall correlation with human scores, is not shown to be out-of-sample. Section 4.5 fits w1–w3 by least squares against human scores on the optimization set, and Section 4.6 computes Pearson on the same M videos. The validation set introduced in Section 3.1 never appears in the results. With 40 videos and 3 fitted weights, a high in-sample correlation is exactly what you'd expect; it doesn't demonstrate alignment with human judgment.\n\nWhat's genuinely new is the integration itself: nobody else, as far as I can tell, has chained these specific components into a single video-editing metric, and the LLM-driven object refinement is a nice touch. The Table 3 breakdown, with each component's correlation, is a useful diagnostic. But the numbers as reported can't be trusted. The fitted weights are never given, so I can't even check whether the sign of w3 compensates for the temporal term. Speaking of which, Section 4.4 writes S_final = ... + w3·(S1−temporal), and the text says subtracting from 1 ensures higher temporal consistency yields a higher final score. As written, it does the opposite unless w3 is negative. That's a load-bearing inconsistency, not a typo — the entire claim about temporal handling rests on it.\n\nThere are also smaller problems: the human evaluation protocol is undocumented (no rater count, no inter-rater agreement, despite the related work calling for Krippendorff's alpha), Table 5 gives per-model human scores but Table 3 correlations presumably use per-video data with no sample size stated, and the 'R1 score' in 4.6 is presumably R² and is never reported. The citations are sloppy — [22] and [11] are mis-attributed — but that's minor.\n\nWho's this for? Anyone building automated video-editing benchmarks. The paper is useful as a cautionary example of fitting-then-claiming, and the component choices are worth copying. But it needs a major revision: report the weights, run a true held-out validation split, correct the temporal formula, and document the human rating protocol. I'd send it to review with the expectation of heavy revision, not desk-reject it — the underlying idea has merit and the code is there for the referees to check.","headline":"Sensible component choices and public code, but the headline correlation is an in-sample fit and the temporal formula contradicts itself; needs major revision, not desk rejection.","tokens_in":11729,"tokens_out":2832,"would_cite":false,"duration_ms":25293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SST-EM claims an automated score can rank edited videos in near-perfect agreement with human quality judgments, reporting a 0.962 Pearson correlation.","keywords":["video editing evaluation","semantic similarity","temporal consistency","object detection","vision-language models","human evaluation correlation","regression weighting","SST-EM"],"falsifier":"Re-run the evaluation on a larger, independently labeled set of edited videos, say at least 100 videos per model with documented inter-rater agreement, and recompute the Pearson, Spearman, and Kendall correlations; if the SST-EM correlations drop substantially or if the fitted weights change dramatically across random halves of the data, the claim that the metric reproduces human judgment at 0.962 would be falsified. A cheaper check is to compute correlations within the 40-video optimization set and see whether the reported 1.000 rank correlations are an artifact of near-tied scores across the handful of editing models.","tokens_in":10691,"feed_emoji":"🎬","tokens_out":6037,"duration_ms":53307,"temperature":0.7,"pith_summary":"The paper proposes SST-EM, an automated score for judging edited videos, and claims it tracks human-quality judgments more closely than existing CLIP-text or EditBoard-style metrics. The metric combines three signals: how well frames semantically match the editing prompt via a vision-language model, whether the primary object stays present via text-conditioned object detection guided by an LLM, and how smoothly consecutive frames change via vision-transformer embedding similarity. These three sub-scores are combined into one weighted formula, with the weights fit by linear regression to human ratings on a small optimization set. The paper reports a Pearson correlation of 0.962 between the final score and human evaluation, with perfect Spearman and Kendall rank agreement, and concludes that a fully automated pipeline can stand in for human assessment of video edits.","feed_headline":"New scoring metric matches human judgment on video edits at 0.962","feed_subtitle":"SST-EM combines prompt understanding, object tracking, and frame smoothness into one automated video-editing score.","key_machinery":"The load-bearing object is the final score formula $$S_{\\mathrm{final}} = w_1 S_{\\mathrm{similarity}} + w_2 S_{\\mathrm{object\\ detection}} + w_3 (1 - S_{\\mathrm{temporal}}),$$ with weights $w_1,w_2,w_3$ obtained by least-squares regression against human evaluation scores. Each component is itself a frame-averaged similarity: PaliGemma captions compared with the editing prompt by cosine similarity, Grounding DINO confidence for the LLM-selected primary object averaged over frames, and cosine similarity between Vision Transformer embeddings of consecutive frames. The $1 - S_{\\mathrm{temporal}}$ term converts high frame-to-frame similarity into a high final score, and the regression step is what turns three generic model outputs into a calibrated approximation of human preference.","core_discovery":"On the paper's own terms, the central discovery is that a composite of three per-frame computations—caption-prompt cosine similarity, text-conditioned object-detection confidence, and consecutive-frame embedding similarity—can be weight-fitted to reproduce human judgments of video-edit quality. The reported headline numbers are a Pearson correlation of 0.962 between the SST-EM final score and human evaluation scores, alongside Spearman and Kendall correlations of 1.000, which the authors read as evidence that the metric ranks edited videos essentially as humans do. The paper also claims the component scores illuminate why the whole works: temporal consistency shows the strongest individual correlation with human judgment, with a 0.927 Pearson correlation and perfect rank agreement, followed by object-detection score at 0.835, while raw context similarity alone is weak at 0.072 Pearson, showing that the temporal term carries much of the load.","pith_inferences":["Editorial inference: the same three-component recipe could be adapted to other video-understanding tasks, such as evaluating text-to-video generation or video summarization, by re-running the regression against whatever human judgments are available for that task.","Editorial inference: weights fitted on 40 videos are likely unstable; a testable extension is to report bootstrap confidence intervals for $w_1,w_2,w_3$ and for the final correlation, which would show how much of the 0.962 reflects the specific optimization sample.","Editorial inference: because the temporal sub-score alone reaches a 0.927 Pearson correlation, a simpler metric built only from consecutive-frame embedding similarity might capture most of the signal, and SST-EM's added complexity would need to be justified on larger datasets with head-to-head comparisons."],"forward_implications":["SST-EM can serve as a drop-in automated replacement for human evaluation in routine video-editing benchmarks, giving a single scalar that ranks models much as humans do.","Because the weights are fitted once on human data, new video-editing models can be compared without rerunning costly human studies, provided the new videos resemble the optimization set.","Researchers can decompose a model's SST-EM score into semantic, object, and temporal components to diagnose why a model ranks low, such as a low temporal sub-score flagging jittery or abrupt edits.","The component-wise correlations suggest future metrics should weight temporal consistency heavily rather than relying only on CLIP-style text-image alignment."],"supporting_citations":[{"why":"Supplies the 40 original-edited video pairs from the Enhanced End-to-End Video Editing dataset used for weight optimization and validation.","marker":"[24]"},{"why":"EditBoard is the source of the human-evaluation sub-scores, such as Imaging Quality, FF-alpha, and Aesthetic Quality, against which SST-EM correlations are computed.","marker":"[7]"},{"why":"PaliGemma is the VLM that produces per-frame captions whose cosine similarity with the editing prompt forms the semantic sub-score.","marker":"[4]"},{"why":"Grounding DINO performs the text-conditioned object detection whose confidence scores form the object-detection sub-score.","marker":"[19]"},{"why":"Demonstrates the regression-to-human-scores approach that SST-EM adopts for weight optimization.","marker":"[22]"},{"why":"Cited for the regression-based combination of metrics against human evaluation and for temporal-dependency modeling in related work.","marker":"[11]"}],"fun_headline_variants":["SST-EM metric matches human video-edit scores at 0.962","Temporal consistency key to SST-EM's 0.962 human correlation","SST-EM: Video-edit scoring that matches human judgment at 0.962","0.962 correlation: SST-EM metric ranks video edits like humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability of the human evaluation scores used as ground truth: the paper reports no rater count, annotation instructions, or inter-rater agreement, and the weights are fitted on only 40 videos, so if those human scores are noisy or biased, every reported correlation inherits that noise.","fun_headline_variants_meta":{"raw":{"variants":["SST-EM metric matches human video-edit scores at 0.962","Temporal consistency key to SST-EM's 0.962 human correlation","SST-EM: Video-edit scoring that matches human judgment at 0.962","0.962 correlation: SST-EM metric ranks video edits like humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3146,"prompt_tokens":938,"completion_tokens":2208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2122}},"tokens_in":554,"tokens_out":2208,"duration_ms":17588,"temperature":1.0,"reasoning_tokens":2122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:38:39.220248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on a larger, independently labeled set of edited videos, say at least 100 videos per model with documented inter-rater agreement, and recompute the Pearson, Spearman, and Kendall correlations; if the SST-EM correlations drop substantially or if the fitted weights change dramatically across random halves of the data, the claim that the metric reproduces human judgment at 0.962 would be falsified. A cheaper check is to compute correlations within the 40-video optimization set and see whether the reported 1.000 rank correlations are an artifact of near-tied scores across the handful of editing models.","supporting_citations":[{"cited_title":"Enhanced end-to-end video editing: Adaptive customization of path, object, and motion dynam- ics","cited_arxiv_id":null,"evidence_quote":"Supplies the 40 original-edited video pairs from the Enhanced End-to-End Video Editing dataset used for weight optimization and validation."},{"cited_title":"EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models","cited_arxiv_id":"2409.09668","evidence_quote":"EditBoard is the source of the human-evaluation sub-scores, such as Imaging Quality, FF-alpha, and Aesthetic Quality, against which SST-EM correlations are computed."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Grounding DINO performs the text-conditioned object detection whose confidence scores form the object-detection sub-score."},{"cited_title":"Gazed–gaze-guided cinematic editing of wide-angle monocular video recordings","cited_arxiv_id":null,"evidence_quote":"Demonstrates the regression-to-human-scores approach that SST-EM adopts for weight optimization."},{"cited_title":"Text-based Talking Video Editing with Cascaded Conditional Diffusion","cited_arxiv_id":"2407.14841","evidence_quote":"Cited for the regression-based combination of metrics against human evaluation and for temporal-dependency modeling in related work."}],"review_version":1}