REVIEW 3 major objections 4 minor 63 references
Video quality scores become far more accurate once models factor in the actual phone screen and lighting of the viewer.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 15:52 UTC pith:3LBZ356N
load-bearing objection Solid multi-device VQA dataset (300+ Android phones + metadata) plus a practical adapter that lifts Kendall on held-out models; the condition-pool claim is plausible but under-ablated. the 3 major comments →
Learning Flexible Generalization in Video Quality Assessment by Bringing Device and Viewing Condition Distributions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Incorporating measured device and viewing-condition metadata into both the aggregation of human pairwise votes and the subsequent adaptation of objective metrics produces substantially higher Kendall rank correlation with real-user preferences than the same metrics used without any condition information. The gain holds across classical and modern full-reference and no-reference metrics and generalises to held-out phone models.
What carries the argument
Blade-Chest aggregation: latent quality scores qi are recovered jointly with two condition-conditioned networks (fc, fb) via EM so that the probability that video i beats j under observed conditions z is given by a sigmoid of the difference of blade-chest distances; a lightweight fully-connected adapter then maps any base-metric score plus a (possibly newly sampled) z vector into a condition-specific quality prediction.
Load-bearing premise
The latent quality numbers recovered by the Blade-Chest EM procedure are treated as true condition-independent scores, and synthetic viewing-condition vectors drawn from a hand-constrained uniform pool are assumed to produce valid soft training targets for the adapter.
What would settle it
Hold out an entire phone model (or an extreme brightness/ambient-light regime) never seen in training, retrain the adapters without any samples from that model or regime, and check whether the adapted metrics still improve Kendall correlation on the held-out votes; if the gain disappears or turns negative, the condition-pool generalisation claim fails.
If this is right
- Streaming platforms can re-rank or re-encode the same video differently for low-end LCD phones versus high-brightness HDR flagships without collecting new subjective data for every device.
- Existing laboratory-trained metrics such as VMAF can be reused rather than retrained from scratch simply by attaching the lightweight condition adapter.
- Banding and other display-sensitive artefacts become measurable under the actual brightness and ambient light of end users instead of under fixed lab conditions.
- Device manufacturers receive a quantitative signal of how their panel technology and peak-brightness choices alter perceived compression quality.
- Future subjective tests can deliberately undersample rare device-condition combinations because the condition pool can later synthesise them.
Where Pith is reading between the lines
- The same adapter architecture could be attached to live QoE estimators inside ABR controllers so that the bitrate ladder itself becomes device-aware in real time.
- Because the condition vector is low-dimensional and hand-crafted, the method may transfer to non-Android platforms once analogous sensor and panel metadata are available.
- The observed drop in perceived quality with larger screen diagonals for fixed VMAF scores supplies a simple, immediately usable rule-of-thumb for mobile-first encoding presets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a large-scale crowd-sourced multi-device VQA dataset of ~250k pairwise preference judgments collected on 300+ Android devices, with rich metadata on display properties (size, resolution, type, brightness) and ambient conditions. It aggregates the votes with a Blade-Chest model (Eq. 1–3, Appendix A) that conditions on a five-dimensional viewing vector z to recover latent quality scores qi, then trains per-metric lightweight MLPs that map an existing VQA prediction plus z to a condition-specific quality estimate. Soft targets for the adapters are generated by sampling synthetic z from a hand-constrained uniform condition pool and passing them through the learned fc/fb networks. On held-out source videos and three frequent phone models, the adapted metrics show substantial Kendall-rank gains over their unadapted counterparts (Fig. 4, Table 3).
Significance. If the claimed condition-aware generalization holds, the work supplies both a uniquely realistic multi-screen mobile dataset and a practical, lightweight adaptation recipe that streaming services could use for device-specific rate-distortion optimization. The public release of data and code is a clear strength. The Blade-Chest aggregation plus synthetic condition pool is a novel and potentially reusable mechanism for sparse multi-condition subjective data. These contributions would meaningfully advance VQA beyond laboratory MOS toward real-world heterogeneous viewing.
major comments (3)
- [Section 6, Fig. 4] Section 6 and Fig. 4: the sole reported performance measure is Kendall rank correlation against the same raw, noisy pairwise votes used to fit Blade-Chest. No correlation of the adapted scores with the recovered qi, with Bradley-Terry aggregates, or with the desktop-monitor reference scores is provided. Because the evaluation target is itself the noisy preference data, it remains unclear whether the large gains reflect genuine condition-aware quality prediction or simply a better ranking of the fixed video set under vote noise.
- [Section 5, Table 3] Section 5 and the condition-pool construction: there is no ablation that freezes, zeros, or randomly permutes the z channel while keeping the soft targets fixed. Without this control it is impossible to isolate how much of the Kendall improvement is attributable to the claimed viewing-condition pathway versus the MLP simply learning a more flexible ranking function from the Blade-Chest soft labels. The OOD phone-model hold-outs (Table 3) still share the identical video content and similar z ranges, so they do not close this gap.
- [Section 5, Limitations] Section 5 and Limitations: the adapters are never compared against the natural baseline of fine-tuning the original VQA models (or their final layers) directly on the proposed dataset without an explicit condition input. Given that the authors themselves note that full retraining was left for future work, the present gains cannot yet be attributed specifically to the multi-screen adaptation strategy rather than to domain adaptation on the new content.
minor comments (4)
- [Abstract, §1] Abstract and Introduction contain repeated grammatical slips (“more than different 300 Android devices”, “First off all we collect”). A careful proof-reading pass is needed.
- [Figure 2] Figure 2 caption and surrounding text mix “LCD/LED” with “HDR/SDR” without clarifying whether the display-type label is self-reported or inferred; a short note on how display technology was obtained would help reproducibility.
- [§4–5] The five-dimensional z vector is described only informally; an explicit listing of the exact features, their normalization, and the hand-picked physical constraints used for the condition pool should appear in the main text or Appendix B.
- [Table 1] Table 1 lists “Proposed … 300+ Crowd. 10,000 240K” while the text claims 250k valid annotations; the discrepancy should be reconciled.
Circularity Check
Minor fitted-input step in the condition-pool soft targets; central Kendall gains remain empirical and not forced by construction.
specific steps
-
fitted input called prediction
[Section 5 (Conditions adaptation) and Figure 3]
"Now, when we have obtained subjective scores qi and trained the networks fc(·) and fb(·) on the real predictions, we can use their combination to expand the training set with simulated samples. ... Conditions z are sampled from a uniform distribution and passed to fc(·) and fb(·) together with a randomly selected pair of distorted versions of the same source video. This yields the probability that video i is of higher quality than video j under the given conditions z. ... The video quality metric predictions for the selected videos and the conditions are processed through the adaptation module"
The soft targets that supervise the adaptation MLP are generated from the already-fitted Blade-Chest latent scores and networks; the adapter therefore learns to reproduce a ranking already extracted from the same pairwise votes. While subsequent Kendall evaluation is performed on raw human votes (not the soft targets) and on held-out content/devices, the training signal itself is a fitted quantity rather than an independent first-principles label, creating a mild fitted-input-to-prediction loop.
full rationale
The paper's derivation is largely self-contained and empirical: a new multi-device pairwise dataset is collected, Blade-Chest (external citation) is fitted via EM to recover latent qi and condition-aware fc/fb, a lightweight MLP adapter is then trained on soft pairwise probabilities generated from those fitted networks under synthetically sampled z, and the resulting adapted metrics are evaluated by Kendall rank correlation against held-out raw human votes (and OOD phone models). No equation equates a reported gain to a fitted constant, no uniqueness theorem is imported from the authors, and the evaluation target is the original noisy votes rather than the soft targets themselves. The only mild circularity is that the adapter's training signal is produced by a model already fitted to the same vote distribution; this is ordinary two-stage fitting rather than a definitional tautology, so the score remains low.
Axiom & Free-Parameter Ledger
free parameters (3)
- adaptation MLP depth and width =
depth=4, width=64
- hand-picked physical constraints on condition pool
- Blade-Chest embedding dimension and network architecture for fc/fb
axioms (3)
- domain assumption Pairwise preference probability is a sigmoid of the difference of Euclidean distances between condition-conditioned blade and chest embeddings (Eq. 1).
- ad hoc to paper A five-dimensional vector (physical size, resolution, brightness, ambient luminance, display type) is a sufficient statistic for the relevant viewing-condition effects.
- domain assumption Latent quality scores qi recovered by EM are comparable across videos and can serve as ground-truth targets once conditions are re-sampled.
invented entities (2)
-
viewing-conditions pool
no independent evidence
-
condition-adaptation module (per-metric MLP)
no independent evidence
read the original abstract
Video quality assessment (VQA) plays a critical role in optimizing video delivery systems. While numerous objective metrics have been proposed to approximate human perception, the perceived quality strongly depends on viewing conditions and display characteristics. Factors such as ambient lighting, display brightness, and resolution significantly influence the visibility of distortions. In this work, we address the question of the multi-screen quality assessment on mobile devices, as this area still tends to be under-covered. We introduce a first large-scale subjective dataset collected across more than different 300 Android devices, accompanied by metadata on viewing conditions and display properties. We propose a strategy for aggregated score extraction and adaptation of VQA models to device-specific quality estimation. Our results demonstrate that incorporating device and context information enables more accurate and flexible quality prediction, offering new opportunities for fine-grained optimization in streaming services. Ultimately, this work advances the development of perceptual quality models that bridge the gap between laboratory evaluations and the diverse conditions of real-world media consumption. We made the dataset and the code available at https://videoprocessing.github.io/device-viewing-conditions.
Figures
Reference graph
Works this paper leans on
-
[1]
264/AVC video quality assessment dataset , author=
MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset , author=. 2016 IEEE international conference on image processing (ICIP) , pages=. 2016 , organization=
2016
-
[2]
Journal of Visual Communication and Image Representation , volume=
VideoSet: A large-scale compressed video quality dataset based on JND measurement , author=. Journal of Visual Communication and Image Representation , volume=. 2017 , publisher=
2017
-
[3]
2016 IEEE international symposium on broadband multimedia systems and broadcasting (BMSB) , pages=
SJTU 4K video subjective quality dataset for content adaptive bit rate estimation without encoding , author=. 2016 IEEE international symposium on broadband multimedia systems and broadcasting (BMSB) , pages=. 2016 , organization=
2016
-
[4]
2018 16th Annual Workshop on Network and Systems Support for Games (NetGames) , pages=
GamingVideoSET: a dataset for gaming video streaming applications , author=. 2018 16th Annual Workshop on Network and Systems Support for Games (NetGames) , pages=. 2018 , organization=
2018
-
[5]
Advances in Neural Information Processing Systems , volume=
Video compression dataset and benchmark of learning-based video-quality metrics , author=. Advances in Neural Information Processing Systems , volume=
-
[6]
2024 16th International Conference on Quality of Multimedia Experience (QoMEX) , pages=
AVT-VQDB-UHD-1-HDR: an open video quality dataset for quality assessment of UHD-1 HDR videos , author=. 2024 16th International Conference on Quality of Multimedia Experience (QoMEX) , pages=. 2024 , organization=
2024
-
[7]
2023 15th International Conference on Quality of Multimedia Experience (QoMEX) , pages=
A subjective dataset for multi-screen video streaming applications , author=. 2023 15th International Conference on Quality of Multimedia Experience (QoMEX) , pages=. 2023 , organization=
2023
-
[8]
2024 IEEE International Conference on Visual Communications and Image Processing (VCIP) , pages=
Multi-Screen Effects on Quality Assessment: Investigating Banding Metrics Inconsistencies , author=. 2024 IEEE International Conference on Visual Communications and Image Processing (VCIP) , pages=. 2024 , organization=
2024
-
[9]
Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
The unreasonable effectiveness of deep features as a perceptual metric , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
-
[10]
IEEE transactions on pattern analysis and machine intelligence , volume=
Image quality assessment: Unifying structure and texture similarity , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2020 , publisher=
2020
-
[11]
2016 , url=
Toward A Practical Perceptual Video Quality Metric , author=. 2016 , url=
2016
-
[12]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
Exploring video quality assessment on user generated contents from aesthetic and technical perspectives , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[13]
IEEE Transactions on Image Processing , volume=
HDR or SDR? A subjective and objective study of scaled and compressed videos , author=. IEEE Transactions on Image Processing , volume=. 2024 , publisher=
2024
-
[14]
IEEE Access , volume=
No-reference video quality estimation based on machine learning for passive gaming video streaming applications , author=. IEEE Access , volume=. 2019 , publisher=
2019
-
[15]
2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) , pages=
UGC-VIDEO: Perceptual quality assessment of user-generated videos , author=. 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) , pages=. 2020 , organization=
2020
-
[16]
2019 IEEE International Symposium on Multimedia (ISM) , pages=
AVT-VQDB-UHD-1: A large scale video quality database for UHD-1 , author=. 2019 IEEE International Symposium on Multimedia (ISM) , pages=. 2019 , organization=
2019
-
[17]
ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
Subjective and objective quality assessment of mobile gaming video , author=. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2022 , organization=
2022
-
[18]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
MD-VQA: Multi-dimensional quality assessment for UGC live videos , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[19]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Kvq: Kwai video quality assessment for short-form videos , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[20]
arXiv preprint arXiv:2507.03990 , year=
LEHA-CVQAD: Dataset To Enable Generalized Video Quality Assessment of Compression Artifacts , author=. arXiv preprint arXiv:2507.03990 , year=
-
[21]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Rich features for perceptual quality assessment of UGC videos , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[22]
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=
Subjective and objective video quality assessment of high dynamic range sports content , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=
-
[23]
2025 , organization =
BrightVQ: Quality Assessment for User-Generated HDR Videos , author =. 2025 , organization =
2025
-
[24]
2022 IEEE international conference on image processing (ICIP) , pages=
Subjective assessment of high dynamic range videos under different ambient conditions , author=. 2022 IEEE international conference on image processing (ICIP) , pages=. 2022 , organization=
2022
-
[25]
arXiv preprint arXiv:2505.21831 , year=
HDRSDR-VQA: A Subjective Video Quality Dataset for HDR and SDR Comparative Evaluation , author=. arXiv preprint arXiv:2505.21831 , year=
-
[26]
IEEE Journal on Emerging and Selected Topics in Circuits and Systems , volume=
Subjective and objective quality assessment of compressed screen content images , author=. IEEE Journal on Emerging and Selected Topics in Circuits and Systems , volume=. 2016 , publisher=
2016
-
[27]
IEEE Transactions on Image Processing , volume=
CVD2014—A database for evaluating no-reference video quality assessment algorithms , author=. IEEE Transactions on Image Processing , volume=. 2016 , publisher=
2016
-
[28]
Journal of Visual Communication and Image Representation , volume=
MCL-V: A streaming video quality assessment database , author=. Journal of Visual Communication and Image Representation , volume=. 2015 , publisher=
2015
-
[29]
IEEE Transactions on Image Processing , volume=
Study of temporal effects on subjective video quality of experience , author=. IEEE Transactions on Image Processing , volume=. 2017 , publisher=
2017
-
[30]
IEEE Network , volume=
A generic quantitative relationship between quality of experience and quality of service , author=. IEEE Network , volume=. 2010 , publisher=
2010
-
[31]
2012 Fourth international workshop on quality of multimedia experience , pages=
The TUM high definition video datasets , author=. 2012 Fourth international workshop on quality of multimedia experience , pages=. 2012 , organization=
2012
-
[32]
IEEE Access , year=
Assessing quality differences for 8K/UHD-2 and 4K/UHD-1 HDR video based on viewing distance , author=. IEEE Access , year=
-
[33]
arXiv preprint arXiv:2401.11485 , year=
ColorVideoVDP: A visual difference predictor for image, video and display distortions , author=. arXiv preprint arXiv:2401.11485 , year=
-
[34]
arXiv preprint arXiv:2304.13625 , year=
HDR-VDP-3: A multi-metric for predicting image differences, quality and contrast distortions in high dynamic range and regular content , author=. arXiv preprint arXiv:2304.13625 , year=
-
[35]
Signal Processing: Image Communication , volume=
HDR-VQM: An objective quality measure for high dynamic range video , author=. Signal Processing: Image Communication , volume=. 2015 , publisher=
2015
-
[36]
2025 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) , pages=
ICME 2025 Generalizable HDR and SDR Video Quality Measurement Grand Challenge , author=. 2025 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) , pages=. 2025 , organization=
2025
-
[37]
The method of paired comparisons , author=
Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=
1952
-
[38]
1978 , publisher=
The Rating of Chessplayers, Past and Present , author=. 1978 , publisher=
1978
-
[39]
Proceedings of the ninth acm international conference on web search and data mining , pages=
Modeling intransitivity in matchup and comparison data , author=. Proceedings of the ninth acm international conference on web search and data mining , pages=
-
[40]
Proceedings of the AAAI conference on artificial intelligence , volume=
Tabnet: Attentive interpretable tabular learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[41]
2021 , howpublished =
Pranav Sodhani , title =. 2021 , howpublished =
2021
-
[42]
IEEE transactions on Image Processing , volume=
FSIM: A feature similarity index for image quality assessment , author=. IEEE transactions on Image Processing , volume=. 2011 , publisher=
2011
-
[43]
2023 , version =
Jon Sneyers , title =. 2023 , version =
2023
-
[44]
Sergey Kastryulin and Dzhamil Zakirov and Denis Prokopenko , year=
-
[45]
IEEE Transactions on image processing , volume=
Image information and visual quality , author=. IEEE Transactions on image processing , volume=. 2006 , publisher=
2006
-
[46]
The thrity-seventh asilomar conference on signals, systems & computers, 2003 , volume=
Multiscale structural similarity for image quality assessment , author=. The thrity-seventh asilomar conference on signals, systems & computers, 2003 , volume=. 2003 , organization=
2003
-
[47]
IEEE Transactions on Image Processing , year=
Chen, Chaofeng and Mo, Jiadi and Hou, Jingwen and Wu, Haoning and Liao, Liang and Sun, Wenxiu and Yan, Qiong and Lin, Weisi , title=. IEEE Transactions on Image Processing , year=
-
[48]
Proceedings of the AAAI conference on artificial intelligence , volume=
Exploring clip for assessing the look and feel of images , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[49]
IEEE Transactions on Circuits and Systems for Video Technology , volume=
Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception , author=. IEEE Transactions on Circuits and Systems for Video Technology , volume=
-
[50]
IEEE Transactions on Image Processing , volume =
Uncertainty-aware blind image quality assessment in the laboratory and wild , author =. IEEE Transactions on Image Processing , volume =
-
[51]
IEEE Transactions on Circuits and Systems for Video Technology , volume=
Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network , author=. IEEE Transactions on Circuits and Systems for Video Technology , volume=
-
[52]
2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) , pages=
Deep learning based full-reference and no-reference quality assessment models for compressed ugc videos , author=. 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) , pages=. 2021 , organization=
2021
-
[53]
IEEE Transactions on Image Processing , year=
UGC-VQA: Benchmarking blind video quality assessment for user generated content , author=. IEEE Transactions on Image Processing , year=
-
[54]
International Journal of Computer Vision , volume=
Unified quality assessment of in-the-wild videos with mixed datasets training , author=. International Journal of Computer Vision , volume=. 2021 , publisher=
2021
-
[55]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[56]
Proceedings of the IEEE international conference on computer vision , pages=
Rankiqa: Learning from rankings for no-reference image quality assessment , author=. Proceedings of the IEEE international conference on computer vision , pages=
-
[57]
IEEE Transactions on Image Processing , volume=
KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment , author=. IEEE Transactions on Image Processing , volume=. 2020 , publisher=
2020
-
[58]
IEEE Transactions on image processing , volume=
No-reference image quality assessment in the spatial domain , author=. IEEE Transactions on image processing , volume=. 2012 , publisher=
2012
-
[59]
Proceedings of the 28th ACM International conference on multimedia , pages=
Norm-in-norm loss with faster convergence and better performance for image quality assessment , author=. Proceedings of the 28th ACM International conference on multimedia , pages=
-
[60]
Proceedings of the 27th ACM international conference on multimedia , pages=
Quality assessment of in-the-wild videos , author=. Proceedings of the 27th ACM international conference on multimedia , pages=
-
[61]
AAAI , year =
Wang, Jianyi and Chan, Kelvin CK and Loy, Chen Change , title =. AAAI , year =
-
[62]
ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
Boosting Image Quality Assessment Performance: Unsupervised Score Fusion by Deep Maximum a Posteriori Estimation , author=. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2024 , organization=
2024
-
[63]
Advances in Neural Information Processing Systems , volume =
A Unified Approach to Interpreting Model Predictions , author =. Advances in Neural Information Processing Systems , volume =. 2017 , publisher =
2017
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.