{"id":"1f6b3837-8397-4edd-88dc-863ffd7c14b5","arxiv_id":"2505.03261","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using an adapted Stable Diffusion model as a spatial feature extractor with a Mamba temporal coherence module, DiffVQA reports top SRCC and PLCC scores on KoNViD-1k, LIVE-VQC, YouTube-UGC, LSVQ, and KVQ, with improved cross-dataset generalization.","lead":"DiffVQA is a no-reference video quality assessment method that adapts a pre-trained Stable Diffusion model into a feature extractor and combines those features with a Mamba-based temporal module to predict video quality scores. The authors report state-of-the-art results on several user-generated content video quality benchmarks and stronger cross-dataset generalization than CNN and ViT backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5's backbone comparison is confounded: the diffusion extractor is adapted on VQA datasets via 30k Controller iterations, while ViT/ResNet/CLIP baselines are only pre-trained; the causal claim about diffusion features is therefore not established.","rationale":"The reader's weakest_assumption identifies the Section 3.2 hypothesis as unsupported because the Controller can copy z0, and its rationale also mentions the missing control for extra adaptation. I agree that the causal interpretation is underdetermined, but I view the backbone-comparison confound as the more load-bearing issue: even if the z0 shortcut were removed, Table 5 still compares an adapted diffusion model against unadapted non-diffusion backbones, so the headline claim 'diffusion model as a feature extractor offers enhanced VQA performance compared to CNN/ViT backbones' is not testable from the reported experiments. The empirical numbers are large and internally consistent, and the engineering contribution (Mamba temporal modeling, feature fusion) is clearly specified, so the appropriate outcome remains conditional acceptance pending a controlled comparison. The proposed test—adapting a ViT with the same budget—would directly settle whether the diffusion prior or the extra adaptation drives the gains. This is why I mark agreement as partial: the reader raised both issues, but I weight the confound more heavily than the z0-copying mechanism.","tokens_in":15394,"tokens_out":5004,"duration_ms":49337,"concrete_test":"Add a matched baseline: take ViT-L/16 (or CLIP ViT-L/14) and give it the same adaptation budget as the diffusion Controller—e.g., train a small ControlNet-style or reconstruction head on the same VQA training frames for 30,000 iterations with the same optimizer and learning rate—then freeze it and train the DiffVQA prediction head under identical settings. If the adapted ViT matches or exceeds DiffVQA's LIVE-VQC SRCC (0.910) and KoNViD-1k cross-dataset SRCC (0.870), the causal claim in the abstract fails; if it remains substantially lower, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that diffusion features offer enhanced VQA performance compared to CNN/ViT backbones—rests on the ablation in Table 5. However, that comparison is confounded by unequal feature-extractor training. Section 4.2 states that the Controller is trained for 30,000 iterations on the targeted VQA datasets before being frozen; the VQA head is then trained on top. The other backbones (ResNet, ViT, CLIP, Vision Mamba) are merely initialized with pre-trained weights and are not given any equivalent dataset-specific adaptation. The paper says all variants are trained under identical conditions, but the diffusion variant includes a large, task-specific adaptation stage that the others lack. The SRCC gap on LIVE-VQC (0.910 for diffusion vs. 0.857 for ViT-L/16) could therefore reflect extra training on the same evaluation distribution rather than the diffusion prior. The VAE* baseline is fine-tuned to reconstruct inputs, which partially controls for adaptation, but it is a much smaller network than the Stable Diffusion U-Net, so capacity and adaptation remain entangled. Additionally, the Controller's access to the clean latent z0 (Eq. 10, Figure 3a) may make the adaptation task nearly trivial, further weakening the claim that the model must extract generalizable semantic and distortion features. For the abstract's causal conclusion to hold, the comparison must hold the adaptation budget fixed across backbones; as reported, the evidence is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiffVQA, a no-reference video quality assessment model that adapts a Stable Diffusion U-Net with a ControlNet-style controller to reconstruct input frames, then uses the adapted model as a frozen feature extractor. Features are obtained from a resized branch (semantics) and a random-crop branch (distortion), fused with a Mamba-based temporal coherence augmentation block, and combined via a weighted MLP to predict video quality scores. Experiments on KoNViD-1k, LIVE-VQC, YouTube-UGC, LSVQ, and KVQ report state-of-the-art intra-dataset SRCC/PLCC and strong cross-dataset generalization, with ablations comparing the diffusion extractor against CNN, ViT, CLIP, Vision Mamba, and VAE backbones.","tokens_in":15733,"tokens_out":4414,"duration_ms":41535,"significance":"If the central causal claim holds, the paper is significant: it demonstrates that a large pre-trained diffusion model can be repurposed as a feature extractor for perceptual quality, with gains of roughly 0.02-0.05 SRCC over prior methods on several benchmarks and notable cross-dataset robustness. The paper also provides a broad backbone comparison and a Mamba-based temporal module, and the experiments are extensive across five datasets. However, the significance hinges on the ablations being able to attribute the gains to the diffusion prior rather than to the extra task-specific adaptation that the diffusion extractor receives; as reported, the evidence is underdetermined.","major_comments":[{"comment":"The comparison in Table 5 is confounded by unequal training budgets. The diffusion feature extractor receives 30,000 iterations of Controller adaptation on the target VQA datasets (Section 4.2), while the ResNet, ViT, CLIP, and Vision Mamba baselines are used only with pre-trained weights. The text states that 'All variants are trained under identical conditions,' but the diffusion variant has an additional task-specific adaptation stage that the other backbones do not. The LIVE-VQC SRCC gap between DiffVQA (0.910) and ViT-L/16 (0.857) could therefore reflect adaptation to the evaluation distribution rather than the diffusion prior. Please add controls that give comparable adaptation to the other backbones (for example, a reconstruction-based or quality-based fine-tuning stage with a similar iteration budget) and also report the performance of the diffusion extractor without the 30,000-iteration adaptation.","section":"Section 4.2, Table 5"},{"comment":"The principal mechanism is asserted rather than derived. In Eq. (10), the Controller receives the concatenation of the noisy latent z_t and the clean latent z_0 as conditioning, so the reconstruction task can be solved almost by copying z_0 through the control pathway. This makes it unclear that faithful reconstruction forces the model to internalize semantic and distortion information. Please provide evidence against an identity shortcut: for example, ablate z_0 conditioning (condition only on z_t or on a corrupted/perturbed z_0), measure reconstruction error on distorted inputs, and show that the extracted features change meaningfully when the input degradation changes.","section":"Section 3.2, Eq. (10)"},{"comment":"The data provenance for the Controller adaptation is underspecified. Section 4.2 says the Controller is trained 'using targeted VQA datasets,' but does not state whether this adaptation is restricted to the training split of each dataset or performed separately for each evaluation scheme. If the Controller adaptation ever uses the test split or the full dataset before splitting, the intra-dataset and cross-dataset numbers in Tables 1-4 would be leakage-affected. Please specify the exact splits used for Controller adaptation and, ideally, include a nested or per-fold adaptation protocol so the reported averages are unbiased.","section":"Section 4.2, Section 4.1"},{"comment":"The claim that the diffusion extractor yields 'clearly separated clusters for different semantic and distortion representations' is supported only by qualitative t-SNE plots on a small synthetic set (10 videos for semantics, 50 for distortion). This is used to justify the semantic/distortion decoupling that underlies the two-branch design. Please provide a quantitative cluster-separation metric (for example, classification accuracy of degradation type or a silhouette score) and a head-to-head comparison with ViT on the same evaluation, so the decoupling claim is not based solely on visual inspection.","section":"Section 5.2, Figure 7"}],"minor_comments":[{"comment":"The reference [17] is used for both KoNViD-1k and LIVE-VQC; LIVE-VQC is a different dataset and should be cited with its own reference (e.g., Ghadiyaram et al., 2018).","section":"Section 4.1"},{"comment":"References [43] and [44] appear to be the same paper, both titled 'High-resolution image synthesis with latent diffusion models,' but they are cited in different contexts and listed with the same venue details; please merge or disambiguate.","section":"References"},{"comment":"The label 'Imgrestoration vs. Imgreconstruction' contains a typo; it should read 'Image restoration vs. Image reconstruction.'","section":"Figure 3(b)"},{"comment":"In Eq. (10), the subscript 'z0' is typeset with a space as 'z 0'; please fix the formatting.","section":"Section 3.2, Eq. (10)"},{"comment":"The methods list in Section 5.1 mentions NIQE, TPQI, and SAQI, but these do not appear in any of the result tables; please either include their results or remove them from the list.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core novelty depends on demonstrating that the diffusion prior, rather than the added adaptation budget or the identity shortcut in the conditioning, is responsible for the reported gains. If the authors can close that gap with the suggested controls, this could be a strong contribution. If the identity-shortcut concern is confirmed and adaptation explains the gains, the contribution would reduce to a fairly standard 'fine-tune the backbone on the target data' result, which is within reach of existing VQA pipelines. I would recommend the editor weight the revision accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: DiffVQA reports a clear state-of-the-art on five VQA datasets by adapting Stable Diffusion into a feature extractor and adding a Mamba temporal module. The engineering is real, and the numbers are impressive—on KoNViD-1k they hit SRCC 0.936 vs. DOVER's 0.909, with similarly large cross-dataset gains. But the abstract's causal claim, that diffusion models make better VQA feature extractors than CNNs or ViTs, is not established by the experiments as reported.\n\nWhat's genuinely new: combining a ControlNet-style adaptation of a large pre-trained diffusion model with a bidirectional Mamba block for temporal modeling in VQA. The architecture is described in enough detail to be reproducible in principle, and the ablations on the TCAB module are useful. The cross-dataset results in Tables 2 and 4 are consistent and well beyond prior methods, so the empirical contribution is substantial if the numbers hold.\n\nThe soft spots are exactly where the stress-test note lands. First, the adaptation loss in Eq. 10 conditions the Controller on the clean latent z0 alongside the noisy z_t. Given both, the model can recover the added noise nearly by arithmetic: z_t minus sqrt(alpha_bar_t) z0, divided by sqrt(1-alpha_bar_t). The reconstruction task is then almost trivial, so the paper's hypothesis that the adapted model 'must extract critical semantic and distortion information' is not supported by the loss. The features may still be useful for VQA, but the paper doesn't show that.\n\nSecond, Table 5's backbone comparison is confounded. The diffusion feature extractor gets 30,000 iterations of adaptation on the target VQA datasets before the VQA head is trained, while the ResNet/ViT/CLIP baselines are only pre-trained and then frozen. The paper says all variants are trained under 'identical conditions,' but that's not accurate—the diffusion variant has a large, task-specific training stage the others lack. The VAE* baseline partially controls for this, but it's a much smaller network, so capacity and adaptation remain entangled. The observed gap could come from the extra training on the evaluation distribution, not from any special property of diffusion models.\n\nThese are not fatal to the empirical results, but they do undercut the abstract's conclusion. The paper deserves serious peer review: the results are strong enough that the community should see them with the confound addressed. I would ask the authors to add a cost-matched ViT or ResNet baseline (adapted on the same VQA data for the same number of iterations), ablate the clean-latent input or justify it theoretically, report standard deviations across their 10 splits, and release code with a commit hash. Until then, I would treat this as an interesting empirical data point rather than evidence that diffusion features are intrinsically better for VQA.","headline":"Strong empirical results on five VQA datasets, but the central claim about diffusion features rests on a confounded backbone comparison and a shortcut-prone adaptation objective.","tokens_in":767,"tokens_out":982,"would_cite":true,"duration_ms":39318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No-reference video quality assessment improves when a diffusion model supplies the features, and the paper reports gains over CNN and ViT backbones on three UGC benchmarks.","keywords":["video quality assessment","no-reference VQA","diffusion feature extractor","Stable Diffusion","Mamba","state space model","user-generated content","cross-dataset generalization"],"falsifier":"Train the identical VQA head and feature-fusion pipeline on features obtained directly from the VAE encoder without the denoising U-Net, or retrain with the Controller given only $\\mathbf{z}_t$ and not $\\mathbf{z}_0$; if the reported correlation scores over ViT backbones persist, the diffusion prior is not the cause, and if they collapse, the reconstruction shortcut is doing the work.","tokens_in":15189,"feed_emoji":"🎬","tokens_out":7430,"duration_ms":67391,"temperature":0.7,"pith_summary":"This paper claims that a latent diffusion model adapted to reconstruct video frames can serve as a feature extractor for no-reference video quality assessment, and that the resulting features align with human quality judgments better than features from CNNs or vision transformers. On LIVE-VQC, KoNViD-1k, and YouTube-UGC, the proposed DiffVQA model reports Spearman correlations of 0.910, 0.936, and 0.923, improving on prior methods by as much as roughly 0.05. The same model also transfers across datasets without fine-tuning, which the authors take as evidence that diffusion priors trained on large image collections carry useful quality-relevant information. If the claim holds, adapting generative models for reconstruction becomes a viable route to perceptual feature extraction, not just synthesis.","feed_headline":"Diffusion features top CNN and ViT for no-reference video quality","feed_subtitle":"Adapted Stable Diffusion features plus Mamba temporal modeling set new best SRCC on three UGC video sets.","key_machinery":"The central object is the adapted diffusion feature extractor: a pre-trained latent diffusion model whose denoising U-Net is augmented by a Controller, trained with the loss $L_{\\text{Adap}}=\\mathbb{E}_{\\mathbf{z}_0,t,\\epsilon}[\\lVert \\epsilon_\\theta(\\mathbf{z}_t,t,\\mathbf{z}_0)-\\epsilon\\rVert^2]$. The load-bearing identity is the reconstruction objective: the paper assumes that if the model reconstructs each input frame closely, its internal features must encode the semantic and distortion information needed for quality scoring. The parallel temporal mechanism is the Temporal Coherence Augmentation Block, built on bidirectional Mamba state-space scans, which supplies long-range temporal coherence features that are merged with the diffusion features before score prediction.","core_discovery":"DiffVQA adapts Stable Diffusion by adding a trainable Controller that takes the noisy latent and the clean latent as input and is trained to denoise the noisy latent back to the original frame; after adaptation, the denoising network's multi-scale features at time step zero plus the reconstructed latent are treated as quality features. A resizing branch yields semantic features and a random-crop branch yields distortion features, and both are fused with temporal-difference features. A parallel bidirectional Mamba block, the Temporal Coherence Augmentation Block, adds long-range temporal coherence features, and the final video score is a learned weighted sum of per-frame quality scores. The paper reports that this setup outperforms CNN, ViT, CLIP, and VAE backbones inside the same architecture, both within datasets and in cross-dataset transfer.","pith_inferences":["A control the paper does not run: remove the clean latent $\\mathbf{z}_0$ from the Controller's input and retrain; if performance drops, the gains come from the copy route rather than from diffusion reconstruction.","Comparing against a frozen, unadapted diffusion feature extractor would separate the contribution of large-scale pretraining from the contribution of the reconstruction adaptation.","The same adapt-then-extract recipe could transfer to other perceptual tasks with small datasets, but the copy-through-conditioning pitfall would need to be controlled first."],"forward_implications":["Diffusion-based feature extractors can replace CNN and ViT backbones in no-reference VQA and improve both same-dataset accuracy and cross-dataset transfer.","The parallel Mamba block offers a stronger temporal modeling alternative to 3D convolutions for long video sequences within the same pipeline.","Splitting features into semantic and distortion branches and learning per-frame weights makes the quality prediction combine spatial content and distortion cues explicitly.","Because the feature extractor transfers across datasets, small VQA datasets can be fine-tuned without sacrificing cross-dataset generalization."],"supporting_citations":[{"why":"Defines Stable Diffusion, the pre-trained latent diffusion model whose denoising network is adapted into the feature extractor.","marker":"[43]"},{"why":"Supplies the Controller mechanism that guides the frozen denoiser to reconstruct the input frame, enabling feature extraction.","marker":"[68]"},{"why":"Introduces the selective state-space model Mamba on which the Temporal Coherence Augmentation Block is built.","marker":"[9]"},{"why":"Provides the spaced-DDPM sampling procedure used at inference to obtain the denoising features at time step zero.","marker":"[40]"},{"why":"Supplies the temporal-difference and frame-feature-fusion design that the TDM and FFF modules adapt.","marker":"[58]"},{"why":"Provides the LSVQ training set and the Patch-VQ baseline; the model is pre-trained on this dataset.","marker":"[66]"},{"why":"Provides KoNViD-1k, one of the benchmark datasets for both intra-dataset and cross-dataset evaluation.","marker":"[17]"},{"why":"Provides the KVQ short-form video dataset that evaluates DiffVQA on a different distribution.","marker":"[34]"}],"fun_headline_variants":["Diffusion features plus Mamba set new VQA benchmarks","Adapted Stable Diffusion features improve video quality prediction","Diffusion model features beat CNN and ViT for VQA","DiffVQA: Diffusion and Mamba for video quality assessment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that because the adapted diffusion model can reconstruct each input frame, its internal features must contain the semantic and distortion information needed for quality scoring; since the clean latent $\\mathbf{z}_0$ is concatenated into the Controller's input, the model may succeed by copying $\\mathbf{z}_0$ rather than by learning generalizable quality features.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion features plus Mamba set new VQA benchmarks","Adapted Stable Diffusion features improve video quality prediction","Diffusion model features beat CNN and ViT for VQA","DiffVQA: Diffusion and Mamba for video quality assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3060,"prompt_tokens":918,"completion_tokens":2142,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2075}},"tokens_in":534,"tokens_out":2142,"duration_ms":16579,"temperature":1.0,"reasoning_tokens":2075,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:56:07.722250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical VQA head and feature-fusion pipeline on features obtained directly from the VAE encoder without the denoising U-Net, or retrain with the Controller given only $\\mathbf{z}_t$ and not $\\mathbf{z}_0$; if the reported correlation scores over ViT backbones persist, the diffusion prior is not the cause, and if they collapse, the reconstruction shortcut is doing the work.","supporting_citations":[{"cited_title":"Improved denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides the spaced-DDPM sampling procedure used at inference to obtain the denoising features at time step zero."},{"cited_title":"Discovqa: Temporal distortion-content transformers for video quality assessment","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal-difference and frame-feature-fusion design that the TDM and FFF modules adapt."},{"cited_title":"Patch-vq:’patching up’the video quality problem","cited_arxiv_id":null,"evidence_quote":"Provides the LSVQ training set and the Patch-VQ baseline; the model is pre-trained on this dataset."},{"cited_title":"The konstanz natural video database (konvid-1k)","cited_arxiv_id":null,"evidence_quote":"Provides KoNViD-1k, one of the benchmark datasets for both intra-dataset and cross-dataset evaluation."},{"cited_title":"Kvq: Kwai video quality assessment for short-form videos","cited_arxiv_id":null,"evidence_quote":"Provides the KVQ short-form video dataset that evaluates DiffVQA on a different distribution."}],"review_version":1}