{"id":"20174170-b7fd-4fb4-9673-20973b5f8cba","arxiv_id":"2501.02706","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MSA-VQA combines CLIP-based prompt checking with cross-attention over frames to predict human quality scores for AI-generated videos, reporting state-of-the-art numbers on the T2VQA-DB benchmark.","lead":"This paper presents MSA-VQA, a model that scores the quality of AI-generated videos by looking at individual frames, short segments, and the whole video. It also uses the text prompt and frame-to-frame changes as extra signals, and reports better correlation with human ratings than earlier methods on one benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Video-level random 9:1 split on 10,000 videos from 27 prompts likely leaks prompt identity into validation; PSS can memorize prompt-quality shortcuts, inflating the SOTA claim.","rationale":"The paper applies CLIP adapters and a multi-level ensemble to AIGC video quality assessment. The single-dataset evaluation is the weak point. Reading Section III-A, T2VQA-DB appears to be built from a small number of generation prompts (27 'subjects') with many videos per prompt; if so, a random 9:1 split does not separate prompts. Because the proposed PSS module is trained with the prompt text as input, the model can use prompt identity as a shortcut to predict quality. That would explain the large PSS ablation gain and would make the comparison with prompt-blind baselines unfair. This is a standard leakage check, not an ad hominem attack. The proposed concrete test settles it: a prompt-disjoint split preserves the PSS contribution only if the semantic-consistency signal is real. I therefore keep the overall CONDITIONAL verdict, but the conditions should include this leakage test in addition to code release and statistical significance reporting.","tokens_in":8950,"tokens_out":5258,"duration_ms":54746,"concrete_test":"Re-run the full protocol with a prompt-disjoint split: partition the 10,000 videos by unique conditional prompt into training and validation sets so no prompt appears in both, then retrain all branches and the PSS/SMA ablations with identical hyperparameters and report SRCC/PLCC on held-out prompts. If the MSA-VQA advantage over T2VQA shrinks or the PSS ablation gain (Table IV, 0.740 to 0.796 average score) largely disappears, the original numbers were inflated by prompt leakage. As a secondary check, train a trivial model that maps only the prompt embedding to the quality score and compare it with the full model under both split regimes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The SOTA claim rests entirely on T2VQA-DB. Section III-A says the dataset contains 10,000 generated videos 'on 27 subjects' and that the authors use a 9:1 random split. If those 27 subjects are the text prompts used to generate the videos, a video-level random split almost certainly places the same prompt in both training and validation. Section II-B then trains the Prompt Semantic Supervision module on the CLIP text encoding of the conditional prompt, with quality labels. Under this split the module need not learn the claimed general rule that larger semantic difference between video and prompt implies lower quality; it can instead memorize a per-prompt quality bias, since each prompt's videos appear on both sides of the split. The strongest baselines (T2VQA, Q-Align, ZOOM-VQA) do not consume the prompt text, so this shortcut is unavailable to them, making the comparison unfair. The paper never states that the split is prompt-disjoint or reports the number of unique prompts in each partition. The PSS ablation gain in Table IV is therefore not evidence for semantic-consistency supervision unless prompt leakage is ruled out. This is a correctness risk for the central claim, not merely a missing baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MSA-VQA, a no-reference quality assessment model specifically for AI-generated videos. It comprises three branches operating at frame, segment, and video levels, each trained with a dedicated loss and combined by a weighted ensemble. Two semantic modules are introduced: a Prompt Semantic Supervision (PSS) module that uses CLIP text encoder adapters to compare video content with the conditional text prompt, and a Semantic Mutation-aware (SMA) module that uses CLIP image features and cross-attention to detect semantic inconsistencies across frames. Experiments on the T2VQA-DB dataset report SRCC 0.810, PLCC 0.825, and average score 0.818, which the authors state is state-of-the-art, surpassing the previous best method T2VQA (SRCC 0.796, PLCC 0.806). Ablations on ensemble strategy and on the two semantic modules are presented in Tables III and IV.","tokens_in":9194,"tokens_out":2799,"duration_ms":29799,"significance":"If the reported results are valid, the paper makes a useful contribution: it demonstrates that explicit semantic supervision from the conditioning prompt can improve AI-generated video quality assessment, and its multilevel ensemble architecture is a reasonable and potentially generalizable design. The reported margin over T2VQA on T2VQA-DB is notable, and the ablation results suggest that both the prompt supervision and the mutation-aware module contribute to the final performance. The main value of the paper is empirical; the claim is falsifiable and the experimental protocol is, in principle, reproducible. However, the significance is currently contingent on the soundness of the train/validation split and on the interpretation of the ablation gains, because the entire result rests on a single dataset and a single split.","major_comments":[{"comment":"","section":"III-A and II-B"},{"comment":"The empirical claim rests on a single random 9:1 split with no error bars, no bootstrap confidence intervals, and no significance tests. The reported gain over T2VQA is SRCC +0.014 and PLCC +0.019, which may be within the noise of the evaluation protocol. Please report results over multiple random splits (ideally with prompt-disjoint partitioning), including standard deviations, or provide a statistical test. This is necessary to support the statement that MSA-VQA 'significantly' outperforms existing methods.","section":"Table II and Section III-A"},{"comment":"The mechanism for propagating video-level quality labels to frame and segment levels is underspecified. Section II-A says that video-level scores are 'distributed across frames' and 'redistributed' (Figure 1), but no formula, temporal weighting scheme, or description of how segment-level labels are derived is given. Since the frame and segment branches are trained with their own losses in Section II-D, the exact target construction is essential for reproducibility and for interpreting the ablation results in Table III. Please specify this redistribution procedure precisely.","section":"Section II-A and II-D"},{"comment":"The paper asserts in Section II-B that 'the greater the semantic difference between a video's content and the conditional prompt used for its generation, the lower its quality score,' but this monotonicity assumption is not directly validated. The only evidence offered is the indirect ablation gain in Table IV on a single dataset, which is also subject to the prompt-leakage concern above. If CLIP text-image alignment does not track human quality judgments for AI-generated distortions, the PSS module could add noise or bias rather than semantic understanding. Please provide direct evidence, such as a correlation analysis between CLIP semantic difference and human quality scores on a held-out set, or a failure-case analysis showing that PSS helps when semantic mismatch is present.","section":"Section II-B and Table IV"}],"minor_comments":[{"comment":"There is an inconsistency in the figure panel references: Section II-B says PSS is 'depicted in Figure 1(c)' and Section II-C says SMA is 'shown in Figure 1(b)', but the caption lists the PSS module as (b) and the SMA module as (c). Please correct the cross-references.","section":"Figure 1 and Sections II-B, II-C"},{"comment":"The notation in Equations (1) and (2) is unclear: Pc1 is the output of adapter g1 applied to the CLS token, then Equation (2) feeds [Pc1, F] into the encoder. Please clarify whether Pc1 is concatenated with the original token sequence, what dimension Pc1 has, and why the projection is applied before the final encoder layer rather than after it.","section":"Equations (1)-(2)"},{"comment":"The phrase '10,000 generated videos on 27 subjects' is ambiguous. It should be clarified whether 'subjects' refers to the text prompts, the underlying scene categories, or some other grouping, since this directly affects the interpretation of the random split and the prompt-leakage risk.","section":"Section III-A"},{"comment":"T2VQA-DB is cited via both [18] and [19]; please disambiguate which reference corresponds to the dataset description and which to the T2VQA metric method, and ensure the citation list matches the text.","section":"Section III-A"},{"comment":"There are several typographical errors, including 'contional' in Section II-B, 'Addtionally' in Section III-C, and inconsistent use of 'Swin Transfomer' in Figure 1. A careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The prompt-leakage concern in Section III-A is the main risk to the paper's central claim. If the random split is not prompt-disjoint, the PSS module can exploit per-prompt quality priors that are unavailable to the baselines, and the entire SOTA comparison would be invalid. I would ask the authors to re-run experiments with a prompt-disjoint split, or at minimum to report the number of shared prompts between training and validation. If the gains disappear under that split, the paper's contribution would be much weaker and the claim should be downgraded accordingly. Additionally, the single-dataset scope and the lack of statistical significance testing should be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a reasonable engineering contribution to AI-generated video quality assessment, with a clean three-level architecture and two CLIP-based modules that improve the numbers on T2VQA-DB. But the headline SOTA claim is shakier than the paper lets on: the 9:1 random split is video-level, and if the 27 'subjects' are the text prompts used to generate the videos (which is the natural reading), then the Prompt Semantic Supervision module can memorize per-prompt quality biases instead of learning a general semantic-consistency signal. That would make the comparison with prompt-blind baselines unfair and the ablation gain not evidence for the stated mechanism.\n\nWhat's genuinely new: the combination of the prompt-supervision branch and the semantic-mutation cross-attention module, applied at frame/segment/video levels and ensembled. The paper does a decent job of situating itself against prior work, and the comparison table is broad. The ablations are helpful and the contributions of each module are visible.\n\nSoft spots beyond the split: no error bars or significance tests so we cannot tell if the 0.01-0.02 SRCC gains are noise; no code or model release; the claim that CLIP semantic distance is monotonically related to quality is asserted from observation and only tested indirectly through one ablation. All of these are fixable. The split issue is the one that could change the conclusion, and it needs a prompt-disjoint split or at least an explicit statement of how many prompts appear in each partition and whether the model's advantage survives when prompts are held out.\n\nThe math and evaluation are standard for the field; the citation pattern is appropriate and builds on the right prior work (SimpleVQA, Zoom-VQA, T2VQA). I don't see fabricated or invented entities.\n\nBottom line: this paper deserves a serious referee, because the idea is relevant and the issue is empirical and addressable. But I would not take the SOTA claim at face value until the split and code questions are answered. For my own reading group, I'd put it on the list to discuss the evaluation design as a case study in leakage.","headline":"Plausible architecture and ablations, but the SOTA claim could be inflated by prompt leakage in the train/validation split; worth reviewing with the split question as the key test.","tokens_in":9727,"tokens_out":1838,"would_cite":false,"duration_ms":18159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MSA-VQA sets a new state of the art for AI-generated video quality assessment, reaching 0.810 SRCC on the T2VQA-DB benchmark by checking generated videos against their text prompts and tracking semantic glitches between frames.","keywords":["AI-generated video quality assessment","no-reference video quality assessment","CLIP semantic supervision","prompt-video alignment","semantic mutation","cross-attention","multilevel feature fusion","T2VQA-DB"],"falsifier":"Build a test set in which video quality is held roughly constant but prompt-video alignment is deliberately varied, or vice versa, and check whether MSA-VQA's PSS-based predictions follow human scores; if the rank correlation over that set is not significantly positive, the monotonicity assumption that PSS rests on fails, and the reported performance gain would not generalize.","tokens_in":8757,"feed_emoji":"🎬","tokens_out":2847,"duration_ms":26856,"temperature":0.7,"pith_summary":"The paper proposes MSA-VQA, a no-reference model that predicts the perceptual quality of AI-generated videos. It argues that quality depends not only on visual distortion but on semantic alignment between the video and the text prompt that generated it, as well as on sudden semantic changes between frames. On the T2VQA-DB dataset, the model reports a Spearman rank correlation of 0.810 and a Pearson correlation of 0.825, beating the previous best method T2VQA (0.796 and 0.806). The authors' key claim is that adding prompt-semantic supervision and a frame-to-frame semantic mutation detector to a multilevel video-quality architecture yields the best current no-reference quality predictor on this benchmark.","feed_headline":"AI-generated video quality check beats prior best on T2VQA-DB","feed_subtitle":"The model checks prompt alignment and frame-to-frame semantic glitches, reaching 0.810 SRCC on T2VQA-DB.","key_machinery":"The central mechanism is a two-part CLIP-based semantic supervision scheme layered onto a multilevel video-quality network. The Prompt Semantic Supervision module adds lightweight adapters to the final two layers of CLIP's text encoder, mapping the class token of the conditional prompt into a quality-aware feature that is injected into each branch. The Semantic Mutation-aware module extracts CLS tokens from CLIP's image encoder on each frame and applies cross-attention between a learnable query and the sequence of frame features, compressing the frame set into a fixed-size representation of semantic change. These modules work with a three-branch ensemble — a Video Swin Transformer for the video level, a Swin Transformer pretrained on LSVQ for the segment level, and a ConvNext-tiny for the frame level — each trained with a different loss and fused through a learned weighted sum.","core_discovery":"MSA-VQA establishes that AI-generated video quality can be predicted more accurately when the model is explicitly supervised with two semantic signals: the consistency between the generated video and its conditional text prompt, and the presence of semantic mutations between consecutive frames. The model processes video at three granularities — frame, segment, and full video — each with a separate backbone and loss, and combines the three predictions with a weighted ensemble. A Prompt Semantic Supervision module adapts CLIP's text encoder to produce a quality-aware prompt feature, while a Semantic Mutation-aware module uses CLIP's image encoder with cross-attention to capture frame-to-frame semantic drift. Systematic ablations show that both modules contribute gains: removing PSS drops the average score from 0.796 to 0.740, and adding SMA raises PLCC from 0.808 to 0.825.","pith_inferences":["A testable extension the paper does not run: swap the CLIP image encoder for a video-text alignment model to see whether temporal semantic drift is captured better, which would isolate the contribution of cross-frame attention from per-frame CLIP features.","The paper leaves implicit that the PSS module's quality-aware prompt feature could be reused as an auxiliary reward signal for guiding text-to-video generation itself, not just for assessment.","Since the ablation evidence for the monotonicity assumption comes from a single dataset, an external check on a held-out set of prompts with intentionally degraded semantic fidelity would be needed to confirm the assumption is not overfitted to T2VQA-DB's generator distribution."],"forward_implications":["If the reported results hold on other AI-generated video datasets, no-reference quality assessment for text-to-video models can be substantially improved by supervising on prompt alignment rather than only on low-level distortion.","The observed monotonic relation — larger semantic difference between prompt and video means lower quality — would give video generation practitioners a cheap, model-based way to filter badly aligned outputs at inference time.","The SMA module's success suggests that sudden semantic mutations, a common failure in diffusion-generated video, are a learnable signal that generalizes beyond the specific generators in T2VQA-DB.","The multilevel ensemble, where each granularity is trained independently with a tailored loss, provides a template for adapting existing UGC video quality models to the AI-generated domain."],"supporting_citations":[{"why":"Supplies the T2VQA-DB dataset and the T2VQA method that is the main baseline and benchmark for the paper's state-of-the-art comparison.","marker":"[19]"},{"why":"Provides the NTIRE 2024 AI-generated content quality assessment challenge context, defining the task and dataset lineage that T2VQA-DB builds on.","marker":"[18]"},{"why":"Supplies the CLIP model whose text and image encoders are the backbone of both the Prompt Semantic Supervision and Semantic Mutation-aware modules.","marker":"[17]"},{"why":"SimpleVQA is the architecture the authors extend, providing the base segment-level Swin Transformer design and the temporal feature idea.","marker":"[14]"},{"why":"Zoom-VQA is the multilevel patches-frames-clips baseline and the source of the insight to integrate image and video features at multiple granularities.","marker":"[15]"},{"why":"The LSVQ dataset is used to pretrain the segment-level Swin Transformer, giving the model real-world UGC quality knowledge before fine-tuning on AI-generated video.","marker":"[26]"}],"fun_headline_variants":["MSA-VQA: semantic-aware model advances AI video quality scores","AI video quality check uses text-prompt and frame-drift signals","Three-level semantic model tops earlier AI-generated video scoring","Prompt and mutation supervision sharpen AI video quality prediction","New MSA-VQA beats prior on AI video quality with semantic clues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The core load-bearing premise is that the semantic difference between a generated video and its text prompt, as measured by CLIP with trained adapters, tracks human quality judgments monotonically, so that a larger measured difference implies lower perceived quality.","fun_headline_variants_meta":{"raw":{"variants":["MSA-VQA: semantic-aware model advances AI video quality scores","AI video quality check uses text-prompt and frame-drift signals","Three-level semantic model tops earlier AI-generated video scoring","Prompt and mutation supervision sharpen AI video quality prediction","New MSA-VQA beats prior on AI video quality with semantic clues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1227,"prompt_tokens":853,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":469,"tokens_out":374,"duration_ms":4495,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:06:15.276736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set in which video quality is held roughly constant but prompt-video alignment is deliberately varied, or vice versa, and check whether MSA-VQA's PSS-based predictions follow human scores; if the rank correlation over that set is not significantly positive, the monotonicity assumption that PSS rests on fails, and the reported performance gain would not generalize.","supporting_citations":[{"cited_title":"Ntire 2024 quality assessment of ai-generated content challenge,","cited_arxiv_id":null,"evidence_quote":"Provides the NTIRE 2024 AI-generated content quality assessment challenge context, defining the task and dataset lineage that T2VQA-DB builds on."},{"cited_title":"A deep learning based no- reference quality assessment model for ugc videos,","cited_arxiv_id":null,"evidence_quote":"SimpleVQA is the architecture the authors extend, providing the base segment-level Swin Transformer design and the temporal feature idea."},{"cited_title":"Zoom-vqa: Patches, frames and clips integration for video quality assessment,","cited_arxiv_id":null,"evidence_quote":"Zoom-VQA is the multilevel patches-frames-clips baseline and the source of the insight to integrate image and video features at multiple granularities."}],"review_version":1}