Pith. sign in

REVIEW 4 major objections 4 minor 34 references

VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A challenge with 127 participants and 1,519 submissions shows how to build face-image quality scorers under a 5-million-parameter cap.

desk verdict A useful challenge report whose submitted text is missing the dataset/eval section and the leaderboard, so the headline results are unverifiable as given. read the letter →

arxiv 2508.18445 v1 pith:46OIKBSH submitted 2025-08-25 cs.CV

Sizhuo Ma , Wei-Ting Chen , Qiang Gao , Jian Wang , Chris Wei Zhou , Wei Sun , Weixia Zhang , Linhan Cao
show 51 more authors
Jun Jia Xiangyang Zhu Dandan Zhu Xiongkuo Min Guangtao Zhai Baoying Chen Xiongwei Xiao Jishen Zeng Wei Wu Tiexuan Lou Yuchen Tan Chunyi Song Zhiwei Xu MohammadAli Hamidi Hadi Amirpour Mingyin Bai Jiawang Du Zhenyu Jiang Zilong Lu Ziguan Cui Zongliang Gan Xinpeng Li Shiqi Jiang Chenhui Li Changbo Wang Weijun Yuan Zhan Li Yihang Chen Yifan Deng Ruting Deng Zhanglu Chen Boyang Yao Shuling Zheng Feng Zhang Zhiheng Fu Abhishek Joshi Aman Agarwal Rakhil Immidisetti Ajay Narasimha Mopidevi Vishwajeet Shukla Hao Yang Ruikun Zhang Liyuan Pan Kaixin Deng Hang Ouyang Fan yang Zhizun Luo Zhuohang Shi Songning Lai Weilin Ruan Yutao Yue
This is my paper · ORCID
classification cs.CV
keywords FaceImageQualityAssessmentNo-ReferenceMeanOpinionScorepredictionlightweightmodelsknowledgedistillationICCVworkshopchallengein-the-wildimagescorrelationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on the VQualA 2025 Challenge on Face Image Quality Assessment, held with the ICCV 2025 workshops. It claims that a large field of participants—127 teams producing 1,519 submissions—can be brought to a common test: predicting human Mean Opinion Scores for face images with realistic blur, noise, compression, and poor lighting, using models no larger than 5 million parameters and 0.5 GFLOPs. Submissions were compared by correlation metrics on an in-the-wild face-image test set, and the report summarizes the main methodological recipes that emerged. If the comparison is sound, it gives the field a practical yardstick for deploying face-quality assessment on constrained devices.

What carries the argument

The challenge itself is the instrument. Named the VQualA 2025 FIQA Challenge, it fixes the conditions under which all methods are judged: a private in-the-wild face-image test set with MOS ground truth, a hard budget of 0.5 GFLOPs and 5 million parameters, and correlation metrics such as Pearson and Spearman coefficients as the score. That shared protocol is what allows heterogeneous architectures and training schemes to be compared as solutions to the same deployment problem.

What would settle it

Take the top-ranked models from this challenge and run them on an independently collected set of face images with fresh human MOS annotations, keeping the same 0.5 GFLOP and 5-million-parameter limits. If correlation scores drop toward chance or the method ordering flips, the challenge's conclusions are tied to its specific test set rather than to face-image quality assessment generally.

Watch

Extended reading notes

Core claim

The paper's central claim is that a constraint-limited challenge can serve as a reliable instrument for measuring and driving progress in practical face-image quality assessment. Under identical limits—0.5 GFLOPs and fewer than 5 million parameters—and a shared in-the-wild test set with human MOS labels, the finalists' methods converge on a small set of design choices: self-training plus knowledge distillation from a teacher trained partly on the target domain, multi-stage progressive training with increasing resolution and stochastic weight averaging, prompt-aware CLIP teachers adapted by LoRA and distilled into MobileNetV3-Small students, and lightweight ensembles supervised by correlation

Load-bearing premise

The rankings and recipes stand or fall on the challenge's single blind test set being large, representative, and honestly annotated; if that dataset is not, the findings apply only to that competition.

Editorial extensions

If this is right

  • Under the shared compute cap, the submitted methods establish that MOS prediction for face images is feasible with lightweight backbones rather than requiring large general-purpose quality models.
  • Teacher-student recipes—CLIP-based prompt-aware teachers or self-trained teachers distilled into MobileNetV3-Small—appear repeatedly in top solutions, pointing to distillation as a default strategy for compact FIQA.
  • Progressive training in stages (increasing input resolution, full-data fine-tuning, weight averaging) and correlation-aware losses (MSE plus Pearson, or ranking losses) are presented as repeatable ingredients for improving agreement with human scores.
  • The challenge's test set and correlation metrics provide a baseline for future work: any new FIQA model can be compared against the documented submissions under the same constraints and score the same way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the constraint budget matches mobile and camera hardware, the winning recipes are a plausible starting point for on-device face-quality gating in photo and video pipelines—an application the paper motivates but does not test.
  • The report compares finalists only on one private test set; a direct extension is to measure how the same models generalize across multiple independently annotated face datasets, especially under mixed degradation types.
  • A concrete way to test whether semantic, face-aware features matter: retrain the top distilled student without the CLIP teacher but with the same data and loss, and compare correlation on hard low-light and occlusion subsets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper is the official report of the VQualA 2025 Challenge on Face Image Quality Assessment (FIQA), held at ICCV 2025 Workshops. It states that participants built lightweight MOS predictors under 0.5 GFLOP and 5M-parameter limits, claims 127 participants and 1519 final submissions, and aims to 'summarize the methodologies and findings' from evaluations using correlation metrics on in-the-wild face images. The present manuscript contains the introduction and method descriptions for several top teams (ECNU-SJTU, MediaForensics, Next, ATHENAFace), with training/testing details, followed by references and affiliation appendices. No dataset/evaluation-protocol section and no quantitative results/leaderboard are present in the provided text.

Significance. The challenge is timely and the efficiency constraints are practically relevant; a trustworthy report would be a useful benchmark for efficient FIQA. The included method descriptions are unusually concrete (architectures, losses, optimizers, schedules, GPU setups), which is a genuine strength for reproducibility. However, because the quantitative evaluation is entirely absent, the paper in its current form does not deliver the promised 'findings.' The significance hinges on whether the missing sections can be supplied.

major comments (4)
  1. [Abstract; Sec. 1 (missing Sec. 2)] The paper asserts 'comprehensive evaluations through correlation metrics on a dataset of in-the-wild face images,' but no section describes the test data or evaluation protocol. There is no dataset card: collection procedure, image count, face-crop/resolution distribution, MOS annotation method, number of annotators, or inter-rater agreement. The metric names (PLCC/SRCC/KRCC) are not even defined. Without these details, the evaluation cannot be reconstructed, which is load-bearing for the central claim.
  2. [Title/Abstract; Secs. 3.1-3.4] No leaderboard or per-team quantitative scores appear anywhere in the manuscript. The report's stated purpose is to summarize 'methods and results,' but the results are missing: no final rankings, no correlation coefficients, no comparison to a baseline, and no explanation of how the 13 included teams were selected among 127 participants. The relative performance claims implied by the organization are therefore unsupported.
  3. [Secs. 3.1, 3.3, 3.4] Several methods depend on 'the provided labeled data' and 'the competition's original dataset,' but the manuscript does not specify this dataset's size, splits, or license. Some teams (Next, ATHENAFace) also use GFIQA-20k externally. The challenge rules regarding allowed external training data and validation-set use are not stated, making it impossible to assess the fairness of comparisons or the risk of overfitting to the test set.
  4. [Sec. 1] The headline efficiency constraints (0.5 GFLOPs, <5M parameters) are never operationalized: no statement of the input resolution at which FLOPs were measured, the counting tool, or whether teacher models were excluded from the parameter budget. Since the challenge's practical relevance depends on these numbers, this is a required part of the protocol.
minor comments (4)
  1. [Sec. 3.2] 'SW A' should be 'SWA'; the section refers to the figure and caption with inconsistent spacing.
  2. [Secs. 3.2-3.4] Several team descriptions refer to figures (Figs. 2-4), but the figures are not included or described beyond minimal captions.
  3. [References] Reference list entries are inconsistent in formatting (e.g., [21], [31], [44] have irregular 'et al.' usage and title capitalization).
  4. [Appendix B] Appendix B (Details about RankCORE) begins in mid-explanation and ends mid-sentence, so the RankCORE method description is incomplete.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: challenge report evaluates held-out MOS predictions; missing results are a completeness issue, not circular reasoning.

full rationale

The paper is a challenge report: it describes a competitive benchmark in which participant models predict Mean Opinion Scores (MOS) for face images and are scored by correlation against held-out human ground truth. No step in the paper fits a parameter to the evaluation labels and then re-predicts that same fit. The method sections describe supervised training on the competition's labeled training split and inference on test images; the evaluation is an external comparison against MOS annotations, not a quantity defined by the model's own training objective. The self-citations (e.g., prior VQualA challenge reports and team-specific prior papers such as [37, 40]) are contextual or methodological descriptions and are not used to establish the challenge outcomes. The main limitation of the submitted text is that the actual scores, test-set description, and evaluation protocol are missing, so the 'findings' cannot be independently verified; however, this is a completeness problem, not circularity. No prediction is equivalent to its input by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the validity of the human MOS labels and the correlation metrics used to rank models. These are standard domain assumptions in IQA research, but their specific application to this dataset cannot be verified from the provided text.

assumptions (3)
  • domain assumption MOS annotations on the test set are a valid ground truth for perceptual face image quality
    The challenge evaluation assumes mean opinion scores from human raters accurately reflect perceptual quality; this underlies all correlation metrics used.
  • domain assumption Correlation metrics (e.g., PLCC, SROCC) are appropriate for ranking FIQA methods
    The abstract claims 'comprehensive evaluations through correlation metrics', which presumes correlation captures model quality.
  • domain assumption The evaluation dataset is representative of in-the-wild face images with arbitrary resolutions
    The abstract states 'a dataset of in-the-wild face images'; generalizability of the findings depends on this representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results." pith.science (2026). https://pith.science/paper/46OIKBSH

@misc{pith2026250818445,
  author       = {Pith},
  title        = {Pith review of: VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46OIKBSH}},
  note         = {Machine review of arXiv:2508.18445}
}
read the original abstract

Face images play a crucial role in numerous applications; however, real-world conditions frequently introduce degradations such as noise, blur, and compression artifacts, affecting overall image quality and hindering subsequent tasks. To address this challenge, we organized the VQualA 2025 Challenge on Face Image Quality Assessment (FIQA) as part of the ICCV 2025 Workshops. Participants created lightweight and efficient models (limited to 0.5 GFLOPs and 5 million parameters) for the prediction of Mean Opinion Scores (MOS) on face images with arbitrary resolutions and realistic degradations. Submissions underwent comprehensive evaluations through correlation metrics on a dataset of in-the-wild face images. This challenge attracted 127 participants, with 1519 final submissions. This report summarizes the methodologies and findings for advancing the development of practical FIQA approaches.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 31 canonical work pages

  1. [1]

    VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results

    Introduction In recent years, face images have become integral to a wide variety of applications, including video communica- tion, photography, augmented reality, and digital content ∗Sizhuo Ma ( sma @ snap . com), Wei-Ting Chen ( weitingchen @ microsoft . com), Qiang Gao ( qgao @ snap . com), Jian Wang (jwang4@snap.com ) and Chris Wei Zhou ( zhouw26@card...

    work page Pith review arXiv 2025

  2. [14]

    Blind image quality assessment using natural scene statistics.IEEE Transactions on Image Processing, 23(1):310–325, 2014

    Lin Kang, Pei Ye, Yubin Li, and David Doermann. Blind image quality assessment using natural scene statistics.IEEE Transactions on Image Processing, 23(1):310–325, 2014. 14

    work page 2014

  3. [15]

    Progressive growing of gans for improved quality, stability, and variation

    Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. 1

    arXiv 2017

  4. [16]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 2, 7

    work page 2015

  5. [17]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 5

    work page 2012

  6. [18]

    Vquala 2025 challenge on engagement prediction for short videos: Methods and results

    Dasong Li, Sizhuo Ma, Hang Hua, Wenjie Li, Jian Wang, Chris Wei Zhou, Fengbin Guan, Xin Li, Zihao Yu, Yiting Lu, Ru-Ling Liao, Yan Ye, Zhibo Chen, Wei Sun, Linhan Cao, Yuqin Cao, Weixia Zhang, Wen Wen, Kaiwei Zhang, Zijian Chen, Fangfang Lu, Xiongkuo Min, Guangtao Zhai, Erjia Xiao, Lingfeng Zhang, Zhenjie Su, Hao Cheng, Yu Liu, Renjing Xu, Long Chen, Xiao...

    work page 2025

  7. [19]

    Vquala 2025 challenge on image super-resolution generated content qual- ity assessment: Methods and results

    Yixiao Li, Xin Li, Chris Wei Zhou, Shuo Xing, Hadi Amir- pour, Xiaoshuai Hao, Guanghui Yue, Baoquan Zhao, Weide Liu, Xiaoyuan Yang, Zhengzhong Tu, and et al. Vquala 2025 challenge on image super-resolution generated content qual- ity assessment: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision (ICCV) Work- shops, pages 1–1...

    work page 2025

  8. [20]

    Ntire 2023 quality assess- ment of video enhancement challenge

    Xiaohong Liu, Xiongkuo Min, Wei Sun, Yulun Zhang, Kai Zhang, Radu Timofte, Guangtao Zhai, Yixuan Gao, Yuqin Cao, Tengchuan Kou, et al. Ntire 2023 quality assess- ment of video enhancement challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1551–1569, 2023. 8

    work page 2023

Show all 34 references
  1. [21]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37:103031–103063, 2024. 6

    2024

  2. [22]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. 2

    2015

  3. [23]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 3

    2021

  4. [24]

    Sgdr: Stochastic gradient descent with restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with restarts. In ICLR, 2016. 4

    2016

  5. [25]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations (ICLR), 2019. 3, 4

    2019

  6. [26]

    Shufflenet v2: Practical guidelines for efficient cnn architec- ture design

    Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018. 5, 6

    2018

  7. [27]

    Edgenext: efficiently amalgamated cnn-transformer architecture for mobile vision applications

    Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muham- mad Anwer, and Fahad Shahbaz Khan. Edgenext: efficiently amalgamated cnn-transformer architecture for mobile vision applications. In European Conference on Computer Vision, pages 3–2...

    2022

  8. [28]

    No-reference image quality assessment in spatial domain

    Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. No-reference image quality assessment in spatial domain. IEEE Transactions on Image Processing, 21(12):4695–4708,

  9. [29]

    Improving road defect detection precision and efficiency with structural pruning techniques

    Hang Ouyang, Tailin Li, Chongru Wang, Yuanting Gu, Fan Yang, and Kaixin Deng. Improving road defect detection precision and efficiency with structural pruning techniques. In 2024 21st International Computer Conference on Wavelet Active Media Technology and Information Processi...

    2024

  10. [30]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalya Gimelshein, Luca Antiga, and Others. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Syst...

    2019

  11. [31]

    Mobilenetv4: universal models for the mobile ecosystem

    Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Ban- bury, Chengxi Ye, Berkin Akin, et al. Mobilenetv4: universal models for the mobile ecosystem. In European Conference on Computer Vision, pages 78–96. Springer, 2024. 7

    2024

  12. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  13. [33]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Designing network design spaces. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10428–10436,

  14. [34]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 1

    2023

  15. [35]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510–4520, 2018. 2, 7, 9, 10

    2018

  16. [36]

    Going the extra mile in face image quality assess- ment: A novel database and model

    Shaolin Su, Hanhe Lin, Vlad Hosu, Oliver Wiedemann, Jin- qiu Sun, Yu Zhu, Hantao Liu, Yanning Zhang, and Dietmar Saupe. Going the extra mile in face image quality assess- ment: A novel database and model. IEEE Transactions on Multimedia, 26:2671–2685, 2023. 1, 2, 7

    2023

  17. [37]

    Ef- ficient face image quality assessment via self-training and knowledge distillation

    Wei Sun, Weixia Zhang, Linhan Cao, Jun Jia, Xiangyang Zhu, Dandan Zhu, Xiongkuo Min, and Guangtao Zhai. Ef- ficient face image quality assessment via self-training and knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision Workshops , pages 1–9,

  18. [38]

    A strong baseline for image and video quality assessment

    Shaoguo Wen and Junle Wang. A strong baseline for image and video quality assessment. arXiv preprint arXiv:2111.07104, 2021. 4

    2021 arXiv

  19. [39]

    Pytorch image models (timm)

    Ross Wightman. Pytorch image models (timm). https: / / github . com / rwightman / pytorch - image - models, 2019. 4

    2019

  20. [40]

    Mspt: A lightweight face image quality assessment method with multi-stage progressive training

    Xiongwei Xiao, Baoying Chen, Jishen Zeng, and Jianquan Yang. Mspt: A lightweight face image quality assessment method with multi-stage progressive training. arXiv preprint arXiv:2508.07590, 2025. 4

    2025 arXiv

  21. [41]

    Self-training with noisy student improves imagenet clas- sification

    Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet clas- sification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10687– 10698, 2020. 3

    2020

  22. [42]

    Rethinking mobile block for ef- ficient attention-based models

    Jiangning Zhang, Xiangtai Li, Jian Li, Liang Liu, Zhucun Xue, Boshen Zhang, Zhengkai Jiang, Tianxin Huang, Yabiao Wang, and Chengjie Wang. Rethinking mobile block for ef- ficient attention-based models. In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 1389–

    2023

  23. [43]

    Emov2: Pushing 5m vision model frontier

    Jiangning Zhang, Teng Hu, Haoyang He, Zhucun Xue, Yabiao Wang, Chengjie Wang, Yong Liu, Xiangtai Li, and Dacheng Tao. Emov2: Pushing 5m vision model frontier. arXiv preprint arXiv:2412.06674, 2024. 10

    2024 arXiv

  24. [44]

    Vquala 2025 challenge on visual quality comparison for large multimodal models: Methods and results

    Hanwei Zhu, Haoning Wu, Zicheng Zhang, Lingyu Zhu, Yixuan Li, Peilin Chen, Shiqi Wang, Chris Wei Zhou, Lin- han Cao, Wei Sun, Xiangyang Zhu, Weixia Zhang, Yucheng Zhu, Jing Liu, Dandan Zhu, Guantao Zhai, Xiongkuo Min, Zhichao Zhang, Xinyue Li, Shubo Xu, Anh Dao, Yifan Li, Hong...

    2025

  25. [45]

    Dynamic convolution-based image dehazing network

    Shi Zhuohang. Dynamic convolution-based image dehazing network. Multimedia Tools and Applications, 83(16):49039– 49056, 2024. 10 A. Teams and Affiliations VQualA 2025 FIQA Track Organizers Members: Sizhuo Ma 1 (sma @ snap . com), Wei-Ting Chen2 (weitingchen@microsoft.com), Qia...

    2024

  26. [1400]

    IEEE Computer Society, 2023. 10

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.