Pith. sign in

REVIEW 4 major objections 4 minor 21 references

Research on Audio-Visual Quality Assessment Dataset and Method for User-Generated Omnidirectional Video

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a 300-clip user-generated omnidirectional video dataset with audio, subjective Mean Opinion Scores, and head-movement traces, along with a no-reference audio-visual quality model that reaches SROCC 0.8245 and PLCC…

desk verdict A genuinely new UGC-ODV audio-visual dataset with a credible subjective experiment, but the SOTA claim for the baseline model is undercut by a confounded comparison protocol. read the letter →

arxiv 2506.10331 v1 pith:4PZCFW4M submitted 2025-06-12 cs.CV eess.IV

Fei Zhao , Da Pan , Zelu Qi , Ping Shi This is my paper · ORCID
classification cs.CVeess.IV
keywords user-generatedomnidirectionalvideoaudio-visualqualityassessmentno-referencedatasetsubjectivescoresheadmovementdatamultimodalfusionmeanopinionscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper constructs the first audio-visual quality assessment dataset for user-generated omnidirectional video: 300 clips captured by five people with consumer 360-degree cameras across ten scene types, with natural video and audio distortions, human Mean Opinion Scores, and head-movement recordings. The central claim is that audio-visual quality of such content is predictable by a no-reference model that combines video features, audio features, and a transformer-based fusion module, and that this model outperforms existing video-only quality measures. A fair-minded reader would care because everyday 360-degree content is growing quickly while existing quality assessment work has concentrated on professionally produced video and has mostly ignored the audio channel.

What carries the argument

The load-bearing mechanism is the paired dataset plus a three-module baseline. The visual branch partitions each equirectangular-projected frame into latitude sub-regions, trains per-region encoders, and applies a backdoor-adjustment weighting scheme to reduce the influence of dimensional confounders on quality features; the audio branch converts the soundtrack into a Mel spectrogram and extracts features with VGGish; and a transformer block with self-attention and cross-attention fuses the two feature streams before a quality regression head. The dataset supplies the subjective ground truth, including 5,026 retained valid opinion scores and more than 12 million head-movement entries, that makes training and comparison possible.

What would settle it

Re-run the four baselines under the same training protocol as the proposed model, with the same optimizer, learning rate, and 80/20 split; if CIQNet or ProVQA then reaches or exceeds SROCC 0.8245, the claimed state-of-the-art result and the attribution of the gain to audio-visual fusion would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that user-generated omnidirectional video quality is better assessed jointly from sight and sound than from video alone. The proposed baseline uses a causal-intervention visual branch, a VGGish-based audio branch, and a transformer fusion module; on the new 300-video dataset it achieves SROCC 0.8245 and PLCC 0.8590, while the same model without the audio branch scores 0.8045 and 0.8254, and replacing the fusion transformer with additive or concatenative merging lowers performance further. The dataset is presented as the first to address audio-visual quality assessment specifically for user-generated omnidirectional content.

Load-bearing premise

The claim that this model is state of the art assumes the comparison against prior methods is fair, yet the proposed model is trained with a higher learning rate than all compared baselines and no sensitivity analysis is reported.

Editorial extensions

If this is right

  • Audio contributes measurably to perceived omnidirectional video quality, since dropping the audio branch lowers SROCC from 0.8245 to 0.8045.
  • The choice of fusion method matters: concatenative and additive fusion both underperform the transformer fusion module in the reported results.
  • Existing 2D user-generated video quality methods transfer poorly to omnidirectional content, with the best 2D baseline reaching only 0.7974 SROCC on this dataset.
  • The dataset provides a shared benchmark with MOS, head movement, and multi-resolution content on which future no-reference audio-visual quality models can be trained and compared.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test, left implicit in the paper, is cross-dataset generalization: fine-tune the model on these 300 clips and evaluate it on professionally captured omnidirectional audio-visual content to see whether the audio-visual advantage persists outside user-generated conditions.
  • The 120 Hz head-movement traces could support viewport-dependent or gaze-contingent quality models, a direction the paper records data for but does not exploit.
  • Because part of the dataset carries four-channel audio, a future audio branch could treat spatial audio directly instead of collapsing the soundtrack to a monaural Mel representation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a new dataset for audio-visual quality assessment of user-generated omnidirectional videos (UGC-ODV), containing 300 sequences captured with two consumer-grade 360-degree cameras, covering 10 scene types, with subjective MOS scores and head-movement data collected from 136 subjects following ITU-T BT.500/P.910 procedures. The authors also propose a no-reference audio-visual quality assessment baseline model consisting of a CIQNet-based video feature extractor, a VGGish-based audio feature extractor, and a transformer-based fusion module. The paper claims that this is the first work on UGC-ODV audio-visual quality assessment and that the proposed model achieves state-of-the-art performance on the proposed dataset.

Significance. The dataset contribution is potentially valuable: it addresses a genuine gap, since prior omnidirectional video quality datasets predominantly focus on PGC content and ignore audio. The subjective experiment is described in reasonable detail, follows established ITU recommendations, and includes outlier screening and SSQ-based dizziness filtering, lending credibility to the MOS ground truth. However, the model's claimed SOTA performance is not currently supported by the reported experiments. The evaluation uses a single random split, unequal training protocols, and a suspiciously identical baseline row in Table 2. If the dataset is released and the evaluation protocol is corrected, the paper could make a useful contribution to the community. The architectural novelty is modest—the main components are borrowed from CIQNet and VGGish—but the audio-visual fusion and the UGC-ODV application are of interest.

major comments (4)
  1. [IV-A.4 and Table 2] The proposed model is trained with Adam at learning rate 1e-3, while DOVER, FastVQA, CIQNet, and ProVQA are fine-tuned with learning rate 3e-5. No sensitivity analysis is reported, so the claimed SOTA margin could be an artifact of unequal training settings. The authors should fine-tune all methods under the same protocol or at least report results across a grid of learning rates and demonstrate that the ranking remains stable.
  2. [Table 2] The CIQNet row (SROCC 0.8045, PLCC 0.8254, KROCC 0.6186, RMSE 0.7198) is numerically identical to the 'Ours (Without Audio)' ablation row. This suggests that the CIQNet comparison is not an independently reproduced baseline but effectively the authors' own video-only branch. Please clarify how the CIQNet result was obtained; if it is an independent reproduction, explain why the numbers coincide exactly. Otherwise, the comparison must be re-run and the SOTA claim re-evaluated.
  3. [IV-A.1 and IV-B] All results are based on a single random 80/20 split of 300 videos, yielding 60 test videos, with no error bars, confidence intervals, or significance tests. The SROCC gap between the proposed model (0.8245) and ProVQA (0.8081) is only about 0.02, which is plausibly within split-to-split variability. The authors should report k-fold cross-validation or multiple random splits with standard deviations, and include a significance test such as paired bootstrap or Wilcoxon signed-rank.
  4. [General (Dataset and Code Availability)] The manuscript contains no statement about releasing the dataset, MOS scores, head movement data, or code, and no cross-dataset evaluation is performed. Since the dataset is the primary claimed contribution, the authors should specify its availability and provide a clear data-release plan; without this, the community cannot independently verify the MOS values or reproduce the model results.
minor comments (4)
  1. [Abstract and Section II] The text contains typographical and formatting inconsistencies: '12,06 million' in Section II should read '12.06 million'; the abstract renders 'A VQA' with a spurious space; and the phrase 'audio-visual' is sometimes hyphenated and sometimes not. Please standardize.
  2. [Table 1] The header row of Table 1 is not fully clear: the 'HM/EM/MOS' column mixes three separate data types, and the D-SA V360 row leaves the MOS entry blank without explanation. Please separate the columns or add a footnote clarifying missing entries.
  3. [Section IV-A.3] The list of compared methods mentions DOVER, FastVQA, CIQNet, and ProVQA, but the ablation rows 'Ours (Without Audio)', 'Ours (Cat)', and 'Ours (Add)' are not introduced in that list. Please describe the ablation variants explicitly in the text.
  4. [Section II-C] The sentence about head movement data says that valid subjective ratings imply valid head movement data, but this assumption is not justified. At minimum, please report the correlation between head movement data quality and subjective ratings or cite prior work supporting this decision.

Circularity Check

1 steps flagged · score 4.0 of 10

SOTA claim is partially self-referential: the CIQNet baseline row in Table 2 is numerically identical to the authors' own no-audio ablation.

  1. other [Section IV-B, Table 2; Section IV-A.3 (Compared Methods)]
    "CIQNet [9] ✓ ✓ ODV 0.8045 0.8254 0.6186 0.7198 ... Ours (Without Audio)* ✓ ✓ ODV 0.8045 0.8254 0.6186 0.7198 ... our model achieves SOTA performance, while the CIQNet [9] and ProVQA [8] show especially serious performance degradation."

    The 'CIQNet [9]' row in Table 2 reproduces all four performance metrics (SROCC 0.8045, PLCC 0.8254, KROCC 0.6186, RMSE 0.7198) of the authors' own 'Ours (Without Audio)' ablation. Because Section III-A states the visual module 'adopt[s] the basic structure of CIQNet', the no-audio ablation is effectively the authors' implementation of the CIQNet backbone trained under the proposed protocol. The comparison in Section IV-B that 'our model achieves SOTA performance, while the CIQNet and ProVQA show especially serious performance degradation' therefore rests, for the CIQNet baseline, on a value produced by the same pipeline as the proposed model. The claimed gain over CIQNet is thus not an independent external benchmark result but an internal ablation contrast (full model vs.

full rationale

The dataset contribution is externally grounded: Mean Opinion Scores come from human raters following ITU-T BT.500, and the model's correlation numbers are computed against those external MOS labels, so the core MOS-prediction claim is not circular. The identified construction-level problem is limited to the SOTA comparison: the CIQNet baseline row is numerically identical to the authors' own no-audio ablation, so the reported margin over CIQNet reduces to an internal ablation contrast rather than an independent benchmark. The remaining comparison methods are external, and the audio-branch benefit is supported by the authors' own ablation even though the fair-comparison protocol is weakened by the 30x learning-rate difference and the single random split. Those latter issues are validity threats, not circularity, and are not counted in the score. Because the central dataset and the model's absolute MOS prediction retain independent content, the score is 4 rather than higher.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard subjective-testing assumptions, pretrained feature transfer, and a single-split evaluation. The dataset's validity depends on the MOS protocol; the model's SOTA claim depends on fair comparison and representative splitting. None of these are proven in the paper, but they are common practice in the field. No new entities are introduced.

free parameters (2)
  • Random 80/20 train/test split = Not specified (no seed given)
    All performance metrics in Table 2 come from one random split. The split is chosen by hand and no variance is reported, so the reported SOTA numbers depend on this specific split.
  • Learning rates = 1e-3 for proposed model; 3e-5 for baselines
    The comparison fairness and the reported gains depend on this hand-chosen hyperparameter discrepancy (Section IV-A).
assumptions (4)
  • domain assumption MOS from the subjective experiment is a reliable ground truth for audio-visual quality.
    The paper assumes ITU-T BT.500/P.910 procedures and SSQ screening yield valid perceptual quality labels. This is standard but unproved in the paper.
  • domain assumption CIQNet visual features pretrained on IQA-ODI transfer to UGC-ODV content.
    The visual branch is initialized with weights trained on omnidirectional images, and the paper provides no analysis of transferability to UGC-ODV.
  • domain assumption VGGish audio features pretrained on YouTube-8M capture the audio quality distortions in the dataset.
    Audio branch uses VGGish pretrained on YouTube-8M; whether this captures UGC audio distortions (wind noise, mechanical noise) is assumed.
  • ad hoc to paper The single held-out split is representative of the dataset distribution.
    The central performance comparison rests on one random split with no cross-validation or seed reporting (Section IV-A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Research on Audio-Visual Quality Assessment Dataset and Method for User-Generated Omnidirectional Video." pith.science (2026). https://pith.science/paper/4PZCFW4M

@misc{pith2026250610331,
  author       = {Pith},
  title        = {Pith review of: Research on Audio-Visual Quality Assessment Dataset and Method for User-Generated Omnidirectional Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PZCFW4M}},
  note         = {Machine review of arXiv:2506.10331}
}
read the original abstract

In response to the rising prominence of the Metaverse, omnidirectional videos (ODVs) have garnered notable interest, gradually shifting from professional-generated content (PGC) to user-generated content (UGC). However, the study of audio-visual quality assessment (AVQA) within ODVs remains limited. To address this, we construct a dataset of UGC omnidirectional audio and video (A/V) content. The videos are captured by five individuals using two different types of omnidirectional cameras, shooting 300 videos covering 10 different scene types. A subjective AVQA experiment is conducted on the dataset to obtain the Mean Opinion Scores (MOSs) of the A/V sequences. After that, to facilitate the development of UGC-ODV AVQA fields, we construct an effective AVQA baseline model on the proposed dataset, of which the baseline model consists of video feature extraction module, audio feature extraction and audio-visual fusion module. The experimental results demonstrate that our model achieves optimal performance on the proposed dataset.

Figures

Figures reproduced from arXiv: 2506.10331 by the authors.

Figure 1
Figure 1. Typical Scene Display of ODV Materials in the Dataset. This figure provides examples of typical scenes from the ODV materials collected in our dataset, showing thumbnails of the typical video materials in ERP format. Additionally, it outlines the content of each scene and the corresponding scene category recorded in the videos. materials encompassing 10 types of scenes. Basic statistical summaries and representative… view at source ↗
Figure 3
Figure 3. Examples of Typical In-the-Wild Video and Audio Distortion Types in This Dataset. This figure shows the typical types of video and audio distortion in the dataset constructed in this paper. These distortions are commonly encountered by users during everyday shooting. range, which corroborates the diverse nature of the A/V content within the dataset. B. Subjective Testing Since there are no established international … view at source ↗
Figure 2
Figure 2. Basic Statistics of Dataset Attributes. This dataset was collected by two men and three women using two omnidirectional shooting devices for both static and dynamic recordings. It includes resolutions of 8K, 6K, and 4K and comprises 300 audio and video materials across ten types of scenes. The 10 types of scenes are: HDR Scene, Dark Scene, Indoor, Outdoor, Night Scene, Garden Architecture, Urban Architecture, Indoor… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Statistical Analysis of Spatial Information (SI) and Temporal Information (TI) in Video Materials. (a) Pie chart of the total number of videos and resolution: shows that the number of video materials at each resolution is sufficient; (b) TI mean histogram: the TI mean …
Figure 5
Figure 5. Figure 5: Layout of the Subjective Experiment Space. The layout positions two locators diagonally, with the subject’s seat centered to maximize locator coverage and improve positioning accuracy. An experimenter’s monitoring area ensures proper guidance and smooth progression of …
Figure 7
Figure 7. Figure 7: MOSs and 95% Confidence Intervals for Each Video [PITH_FULL_IMAGE:figures/full_fig_p003_7.png]
Figure 6
Figure 6. Figure 6: Schematic Diagram of Subjective Experiment Process. Pre-training: The subjects sit in a swivel chair and get used to the viewing environment and equipment. They then wear the display device and learn to use the handle. If they feel uncomfortable, the test is terminated…
Figure 8
Figure 8. Figure 8: Model Architecture. It includes visual feature extraction module, audio feature extraction module and audio-visual feature fusion module. TABLE 2 MODEL PERFORMANCE COMPARISON TABLE. Approaches Attributes Performance on Our Dataset No reference Finetuned Type1 With Audi…
Figure 9
Figure 9. Figure 9: Scatter Plots of the MOS and Prediction. A four-order polynomial nonlinear fitting obtains the curves. The brightness of scatter points from dark to bright means density from low to high. seen from [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

  1. [1]

    State-of-the-art in 360 video/image processing: Perception, assessment and compression,

    M. Xu, C. Li, S. Zhang, and P. Le Callet, “State-of-the-art in 360 video/image processing: Perception, assessment and compression,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 1, pp. 5–26, 2020

    work page 2020

  2. [2]

    Perceptual quality assessment of virtual reality videos in the wild,

    W. Wen, M. Li, Y . Yao, X. Sui, Y . Zhang, L. Lan, Y . Fang, and K. Ma, “Perceptual quality assessment of virtual reality videos in the wild,” IEEE Transactions on Circuits and Systems for Video Technology, 2024

    work page 2024

  3. [3]

    The effect of audio on the experience in virtual reality: a scoping review,

    I. d. V . Bosman, O. . Buruk, K. Jørgensen, and J. Hamari, “The effect of audio on the experience in virtual reality: a scoping review,” Behaviour & Information Technology, vol. 43, no. 1, pp. 165–199, 2024

    work page 2024

  4. [4]

    Bridge the gap between vqa and human behavior on omnidirectional video: A large-scale dataset and a deep learning model,

    C. Li, M. Xu, X. Du, and Z. Wang, “Bridge the gap between vqa and human behavior on omnidirectional video: A large-scale dataset and a deep learning model,” in Proceedings of the 26th ACM international conference on Multimedia , 2018, pp. 932–940

    work page 2018

  5. [5]

    Ssv360: A dataset on subjetive quality assessment of 360 videos for standing and seated viewing on an hmd,

    M. Elwardy, H.-J. Zepernick, and Y . Hu, “Ssv360: A dataset on subjetive quality assessment of 360 videos for standing and seated viewing on an hmd,” in 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) . IEEE, 2022, pp. 01–06

    work page 2022

  6. [6]

    D-sav360: A dataset of gaze scanpaths on 360 ambisonic videos,

    E. Bernal-Berdun, D. Martin, S. Malpica, P. J. Perez, D. Gutierrez, B. Masia, and A. Serrano, “D-sav360: A dataset of gaze scanpaths on 360 ambisonic videos,” IEEE Transactions on Visualization and Computer Graphics, 2023

    work page 2023

  7. [7]

    Perceptual quality assessment of omnidirectional audio-visual signals,

    X. Zhu, H. Duan, Y . Cao, Y . Zhu, Y . Zhu, J. Liu, L. Chen, X. Min, and G. Zhai, “Perceptual quality assessment of omnidirectional audio-visual signals,” in CAAI International Conference on Artificial Intelligence . Springer, 2023, pp. 512–525

    work page 2023

  8. [8]

    Blind vqa on 360 video via progressively learning from pixels, frames, and video,

    L. Yang, M. Xu, S. Li, Y . Guo, and Z. Wang, “Blind vqa on 360 video via progressively learning from pixels, frames, and video,” IEEE Transactions on Image Processing , vol. 32, pp. 128–143, 2022

    work page 2022

Show all 21 references
  1. [9]

    Omnidirectional video quality assessment with causal intervention,

    Z. Hu, L. Liu, and Q. Sang, “Omnidirectional video quality assessment with causal intervention,” IEEE Transactions on Broadcasting , 2024

    2024

  2. [10]

    ITU, 2008

    ITU, Subjective video quality assessment methods for multimedia appli- cations. ITU, 2008

    2008

  3. [11]

    ITU, 2012

    ITU, Methodology for the subjective assessment of the quality of television pictures. ITU, 2012

    2012

  4. [12]

    Simulator sickness questionnaire: twenty years later,

    S. A. Balk, D. B. Bertola, and V . W. Inman, “Simulator sickness questionnaire: twenty years later,” in Driving Assessment Conference , vol. 7, no. 2013. University of Iowa, 2013

    2013

  5. [13]

    IPQ, “Ipq,” https://igroup.org/pq/ipq/index.php, 1 2016

    2016

  6. [14]

    Viewport-based cnn: A multi-task approach for assessing 360 video quality,

    M. Xu, L. Jiang, C. Li, Z. Wang, and X. Tao, “Viewport-based cnn: A multi-task approach for assessing 360 video quality,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 4, pp. 2198– 2215, 2020

    2020

  7. [15]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,

    H. Wu, E. Zhang, L. Liao, C. Chen, J. Hou, A. Wang, W. Sun, Q. Yan, and W. Lin, “Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 20 ...

    2023

  8. [16]

    Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,

    H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, “Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,” in European conference on computer vision. Springer, 2022, pp. 538–554

    2022

  9. [17]

    The robust feature extraction of audio signal by using vggish model,

    M. Diwakar and B. Gupta, “The robust feature extraction of audio signal by using vggish model,” 2023

    2023

  10. [18]

    Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline,

    T. Geng, T. Wang, J. Duan, R. Cong, and F. Zheng, “Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 942–22 951

    2023

  11. [19]

    Catr: Combinatorial- dependence audio-queried transformer for audio-visual video segmen- tation,

    K. Li, Z. Yang, L. Chen, Y . Yang, and J. Xiao, “Catr: Combinatorial- dependence audio-queried transformer for audio-visual video segmen- tation,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 1485–1494

    2023

  12. [20]

    Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images,

    L. Yang, M. Xu, X. Deng, and B. Feng, “Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images,” in 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021, pp. 1–6

    2021

  13. [21]

    Youtube-8m: A large-scale video classi- fication benchmark,

    S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadara- jan, and S. Vijayanarasimhan, “Youtube-8m: A large-scale video classi- fication benchmark,” arXiv preprint arXiv:1609.08675 , 2016

    2016 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.