REVIEW 4 major objections 6 minor 1 cited by
Acquire Precise and Comparable Fundus Image Quality Score: FTHNet and FQS Dataset
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Model scores fundus image quality at 0.94 correlation
desk verdict FQS dataset is a genuinely useful resource in principle, but the paper's own tables contradict its headline numbers and the statistical claim is unsupported as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is FTHNet, a four-part architecture: a shifted-window transformer backbone that extracts multi-resolution features; a Distortion Perception Network that condenses each resolution into a compact vector; a Parameter Hypernetwork that reads the deepest feature map and generates the weight and bias parameters; and a five-layer Target Network that maps the condensed distortion vectors to a score in [0,100]. The hypernetwork does the central work: because it produces the regressor's parameters from the input image itself, the scoring function can adapt to the particular mix of blur, haze, uneven illumination, and over-darkness in each fundus image. The paper's ablation supports this role: replacing the hypernetwork with directly learned parameters drops SRCC from 0.9358 to 0.6092. The shifted-window attention in the backbone keeps computational cost linear in spatial size, which is how the model stays light enough for the reported sub-60 ms inference times.
What would settle it
Re-run FTHNet's evaluation with the folds regrouped so that every image of the same patient or eye lies in a single training or test partition; if SRCC or PLCC falls substantially below the reported roughly 0.94, the claimed generalization is inflated by patient-level leakage.
Extended reading notes
Core claim
The paper's central claim is that fundus image quality can be predicted as a continuous score by a lightweight network that combines a shifted-window transformer backbone with a hypernetwork that generates the weights of a small regression head from the image's own features. The FQS dataset supplies the ground truth: each of 2,246 fundus images carries a mean opinion score between 0 and 100 obtained as a weighted average of six ophthalmologists' ratings, plus a Good/Usable/Reject label. On this dataset the paper reports Pearson correlation 0.9423 and Spearman correlation 0.9488 between predicted and human-assigned scores, saying FTHNet outperforms the compared model-based and deep-learning IQA methods while using fewer parameters and less computation. The authors also report that ablation results show removing the hypernetwork from FTHNet drops SRCC from 0.9358 to 0.6092, and that deployed as an API the model scores a single fundus image in about 45-56 ms.
Load-bearing premise
The load-bearing premise is that the random 80/15/5 splits keep all images from the same patient and the same eye in a single partition, a condition the paper does not explicitly confirm; if violated, the reported held-out correlations would be inflated.
Editorial extensions
If this is right
- If the reported correlations hold outside the FQS splits, a continuous quality score gives hospitals an objective threshold for whether a fundus image is acceptable for diagnosis, instead of relying on broad category labels.
- The small model sizes (5.66M and 14.88M parameters) and sub-60 ms per-image inference would allow quality checking to run during acquisition, so clinicians can immediately re-shoot unreadable images.
- Because FTHNet's regression head is generated per image by the hypernetwork, the same architecture could be adapted to quality scoring of other clinical image types with diverse distortions.
- The FQS dataset, with continuous MOS and three-level labels from multiple ophthalmologists, provides a common test bed for other FIQA algorithms to be retrained and compared on the same scale.
Reading between the lines
- A test the paper leaves implicit is cross-device generalization: retraining on its two capture devices and testing on fundus images from other cameras would show whether the 0.94 correlation is tied to specific equipment or reflects a device-invariant notion of clinical quality.
- The labeling rubric explicitly weights visibility of the macula, optic disc, and vessel structure, so FTHNet's score may be a better proxy for diagnostic usability than generic IQA measures; one could test that by measuring whether a downstream disease detector's errors concentrate in images FTHNet scores low.
- The FQS annotation protocol, including a reference set, weighted senior and junior ratings, and standard-deviation monitoring, could be transferred to build continuous quality labels for other imaging modalities such as OCT, where similar clinical-quality needs exist.
- If the hypernetwork genuinely adapts to distortion type, feeding the same image with different synthetic distortions should move the generated target-network parameters in clusterable, distortion-specific ways; checking that would give an interpretable handle on what drives the score.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a fundus image quality assessment (FIQA) dataset, FQS, containing 2246 fundus images with continuous MOS and three-level quality labels, and proposes FTHNet, a Transformer-based hypernetwork that predicts quality scores. The authors report that FTHNet outperforms several existing IQA methods with fewer parameters and less computation, and they provide deployment experiments and inference time measurements. Evaluation is performed on the authors' own dataset with random 80/15/5 splits repeated ten times.
Significance. If the reported performance were reliable, the FQS dataset and the lightweight FTHNet architecture would be a useful contribution to FIQA, where continuous quality scores are still uncommon and real-time deployment is clinically relevant. The authors are to be credited for constructing a clinical dataset with a documented labeling protocol and for reporting deployment latency. However, the central claims are not currently substantiated: the abstract's headline numbers do not appear in the results tables, the improvement claims in the text do not match Table 1, and the evaluation protocol is described inconsistently. As written, the significance cannot be assessed beyond the potential value of the dataset itself.
major comments (4)
- [Abstract; Tables 1-3] The abstract reports PLCC 0.9423 and SRCC 0.9488 for FTHNet, but these values do not appear anywhere in Tables 1, 2, or 3. Table 1 reports FTHNet-S with PLCC 0.9435 and SRCC 0.9330, and FTHNet-L with PLCC 0.9442 and SRCC 0.9358. Since the central performance claim in the abstract is not reproducible from the paper's own reported results, the headline result is unsupported.
- [Section 3.2 vs Table 1] The text states that FTHNet-L achieves PLCC improvements of 0.0133, 0.0183, 0.5161, and 0.6690 over GraphIQA, HyperIQA, TRIQ, and DeepIQA, respectively. Computing the same differences from Table 1 gives 0.0087, 0.0137, 0.5115, and 0.6644. The claimed SRCC improvements (0.0143, 0.0072, 0.5270, 0.6459) also differ from the table-derived values (0.0078, 0.0007, 0.5205, 0.6394). The reported advantage is therefore not backed by the tabulated data.
- [Section 2.2.1, Section 3.3.3, Tables 1 and 3] The identity of the proposed models is inconsistent. Section 2.2.1 defines FTHNet-L as (2,4,6,2) with C=64, but Section 3.3.3 states that FTHNet-L is (2,2,6,2) with C=64, and Table 3 lists both configurations with different results. Moreover, Table 1's FTHNet-S (SRCC 0.9330, PLCC 0.9435) differs from Table 3's row for the identical configuration (2,4,6,2)/C32 (SRCC 0.9303, PLCC 0.9424). It is unclear which configuration actually produced the reported results.
- [Abstract vs Section 3.1; Sections 2.1.1 and 3.2] The abstract claims that all experiments used 10-fold cross-validation, but Section 3.1 describes a random 80/15/5 split repeated for 10 rounds, which is not 10-fold cross-validation. No per-round results or error bars are reported, so the assertion of statistical significance in Section 3.2 is not supported. In addition, Section 2.1.1 and Section 3.1 do not state whether multiple images from the same eye or patient were grouped into the same split, leaving open the possibility of patient-level leakage that could inflate the reported correlations.
minor comments (6)
- [Eq. (1) and following text] The text says lambda1 and lambda2 represent the weights of junior and experienced ophthalmologists, but the equation multiplies Oi (the experienced scores) by lambda1 and Oji (the junior scores) by lambda2, so the assignment appears reversed and should be corrected.
- [Figure 1 caption] The caption contains a typo: 'Exapmples' should be 'Examples'.
- [Table 2] The loss ablation includes an 'L1+L2' row that is not described in the text, and the formatting of 'L smoothL1' is inconsistent with the other rows.
- [Section 3.2 vs Table 1] The text reports 5.67 M parameters for FTHNet-S, while Table 1 lists 5.662 M and Table 3 lists 5.558 M for the configuration described as FTHNet-S; these numbers should be reconciled.
- [Section 3.2] The phrase 'significantly outperforming' is used without any significance test or confidence interval; please either provide statistical evidence or use weaker wording.
- [Section 4.1] The deployment experiment is anecdotal and does not quantify clinical utility, for example by measuring agreement between the system's quality scores and clinicians' decisions on a separate validation set.
Circularity Check
No significant circularity: the FQS-trained regressor is evaluated on held-out images, and no load-bearing claim reduces to its own inputs by construction.
full rationale
The paper's central derivation is a supervised regression model trained on the FQS labels and evaluated on held-out images, so the reported PLCC/SRCC values are not defined in terms of the model's inputs or fitted parameters. The FQS MOS labels come from a human weighted-average labeling process (Eq. 1), and FTHNet predictions are trained outputs, not algebraic rearrangements of those labels. The repeated random 80/15/5 split in Section 3.1 provides a genuine held-out evaluation, even though the abstract's phrase '10-fold cross-validation' is inconsistent with that described protocol. Baselines are retrained on the same FQS data, and no load-bearing result is imported from the authors' own prior work; the architectural components (Swin-style W-MSA, HyperIQA-style hypernetwork) are cited external methods. The internal mismatches between the abstract's PLCC 0.9423/SRCC 0.9488, the Section 3.2 improvement claims, and the values in Table 1, as well as the unstable model configuration labels between Section 2.2.1, Table 3, and Section 3.3.3, are serious consistency and statistical-reporting defects that undermine the 'significantly outperforming' claim, but they are not circularity: no equation or construction forces the reported numbers to equal their inputs. Selecting the best configuration from many ablations on the same evaluation protocol can inflate apparent performance, but that is selection bias rather than a prediction that reduces by construction to a fitted quantity. The paper itself acknowledges limitations in Section 4.4 (failed cases and sample-size limits), which further shows the evaluation is empirical rather than definitional. Therefore, no circular step meeting the required evidentiary standard is present.
Assumptions & free parameters
free parameters (3)
- MOS weighting coefficients lambda1 and lambda2 =
lambda1=0.11, lambda2=0.22
- Training hyperparameters (learning rate, batch size, iterations, model depth/width) =
lr=5e-5, batch=16, 120000 iterations (853 epochs), FTHNet-L (2,2,6,2), C=64
- Reference set size and scoring rubric =
330 images, five degradation levels per category
assumptions (4)
- domain assumption The weighted average of six ophthalmologists' scores is a valid continuous ground truth for fundus image quality.
- domain assumption Images are independent across train/test splits (no patient or eye-level leakage).
- domain assumption Performance on the FQS dataset, split randomly, generalizes to other fundus cameras, populations, and clinical settings.
- domain assumption Standard IQA metrics (PLCC, SRCC, RMSE) adequately capture clinically meaningful quality.
invented entities (2)
-
FQS dataset
-
FTHNet architecture
Cite this review
Pith. "Pith review of Acquire Precise and Comparable Fundus Image Quality Score: FTHNet and FQS Dataset." pith.science (2026). https://pith.science/paper/ASPZVDIW
@misc{pith2026241112273,
author = {Pith},
title = {Pith review of: Acquire Precise and Comparable Fundus Image Quality Score: FTHNet and FQS Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/ASPZVDIW}},
note = {Machine review of arXiv:2411.12273}
}
read the original abstract
The retinal fundus images are utilized extensively in the diagnosis, and their quality can directly affect the diagnosis results. However, due to the insufficient dataset and algorithm application, current fundus image quality assessment (FIQA) methods are not powerful enough to meet ophthalmologists` demands. In this paper, we address the limitations of datasets and algorithms in FIQA. First, we establish a new FIQA dataset, Fundus Quality Score(FQS), which includes 2246 fundus images with two labels: a continuous Mean Opinion Score varying from 0 to 100 and a three-level quality label. Then, we propose a FIQA Transformer-based Hypernetwork (FTHNet) to solve these tasks with regression results rather than classification results in conventional FIQA works. The FTHNet is optimized for the FIQA tasks with extensive experiments. Results on our FQS dataset show that the FTHNet can give quality scores for fundus images with PLCC of 0.9423 and SRCC of 0.9488, significantly outperforming other methods with fewer parameters and less computation complexity.We successfully build a dataset and model addressing the problems of current FIQA methods. Furthermore, the model deployment experiments demonstrate its potential in automatic medical image quality control. All experiments are carried out with 10-fold cross-validation to ensure the significance of the results.
Figures
Forward citations
Cited by 1 Pith paper
-
FundaQ-8: A Clinically-Inspired Scoring Framework for Automated Fundus Image Quality Assessment
An eight-attribute fundus quality rubric with a ResNet18 regressor, plus a weakly supported claim that quality-aware filtering improves DR grading.
Reference graph
Works this paper leans on
-
[1]
Multitasking Deep Learning Model for Detection of Five Stages of Diabetic Retinopathy
S. Majumder, N. Kehtarnavaz, Multitasking deep learning model for detection of five stages of diabetic retinopathy, arXiv preprint arXiv:2103.04207
-
[2]
C.-H. Hua, K. Kim, T. Huynh-The, J. I. You, S.-Y . Yu, T. Le- Tien, S.-H. Bae, S. Lee, Convolutional network with twofold feature augmentation for diabetic retinopathy recognition from multi-modal images, IEEE Journal of Biomedical and Health Informatics
-
[3]
Y . Peng, S. Dharssi, Q. Chen, T. D. Keenan, E. Agr ´on, W. T. Wong, E. Y . Chew, Z. Lu, Deepseenet: a deep learning model for automated classification of patient-based age-related macu- lar degeneration severity from color fundus photographs, Oph- thalmology 126 (4) (2019) 565–575
work page 2019
-
[4]
P. Burlina, D. E. Freund, N. Joshi, Y . Wolfson, N. M. Bressler, Detection of age-related macular degeneration via deep learn- ing, in: 2016 IEEE 13th International Symposium on Biomedi- cal Imaging (ISBI), IEEE, 2016, pp. 184–188
work page 2016
-
[5]
C. M. G. Cheung, T. Y . Lai, P. Ruamviboonsuk, S.-J. Chen, Y . Chen, K. B. Freund, F. Gomi, A. H. Koh, W.-K. Lee, T. Y . Wong, Polypoidal choroidal vasculopathy: definition, patho- genesis, diagnosis, and management, Ophthalmology 125 (5) (2018) 708–724
work page 2018
- [6]
-
[7]
W. Liao, B. Zou, R. Zhao, Y . Chen, Z. He, M. Zhou, Clini- cal interpretable deep learning model for glaucoma diagnosis, IEEE journal of biomedical and health informatics 24 (5) (2019) 1405–1412
work page 2019
- [8]
Show all 32 references
-
[9]
S. C. Lee, Y . Wang, Automatic retinal image quality assessment and enhancement, in: Medical imaging 1999: image processing, V ol. 3661, SPIE, 1999, pp. 1581–1590
1999
-
[10]
Lalonde, L
M. Lalonde, L. Gagnon, M.-C. Boucher, et al., Automatic visual quality assessment in optical fundus images, in: Proceedings of vision interface, V ol. 32, Ottawa, 2001, pp. 259–264
2001
-
[11]
K ¨ohler, A
T. K ¨ohler, A. Budai, M. F. Kraus, J. Odstr ˇcilik, G. Michelson, J. Hornegger, Automatic no-reference quality assessment for retinal fundus images using vessel segmentation, in: Proceed- ings of the 26th IEEE international symposium on computer- based medical systems, IEEE, ...
2013
-
[12]
Bosse, D
S. Bosse, D. Maniry, K.-R. M ¨uller, T. Wiegand, W. Samek, Deep neural networks for no-reference and full-reference image qual- ity assessment, IEEE Transactions on Image Processing 27 (1) (2018) 206–219. doi:10.1109/TIP.2017.2760518
2018
-
[13]
S. Su, Q. Yan, Y . Zhu, C. Zhang, X. Ge, J. Sun, Y . Zhang, Blindly assess image quality in the wild guided by a self- adaptive hyper network, in: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2020, pp. 3667–3676
2020
-
[14]
S. Sun, T. Yu, J. Xu, W. Zhou, Z. Chen, Graphiqa: Learning distortion graph representations for blind image quality assess- ment, IEEE Transactions on Multimedia
-
[15]
J. You, J. Korhonen, Transformer for image quality assessment, in: 2021 IEEE International Conference on Image Process- ing (ICIP), 2021, pp. 1389–1393. doi:10.1109/ICIP42928. 2021.9506075
2021
-
[16]
H. Fu, B. Wang, J. Shen, S. Cui, Y . Xu, J. Liu, L. Shao, Evalua- tion of retinal image quality assessment networks in di fferent color-spaces, in: D. Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, A. Khan (Eds.), Medical Im- age Computing and Computer...
2019
-
[17]
Y . Shen, R. Fang, B. Sheng, L. Dai, H. Li, J. Qin, Q. Wu, W. Jia, Multi-task fundus image quality assessment via transfer learn- ing and landmarks detection, in: International Workshop on Ma- 10 chine Learning in Medical Imaging, Springer, 2018, pp. 28–36
2018
-
[18]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008
2017
-
[19]
A. Raj, A. K. Tiwari, M. G. Martini, Fundus image quality as- sessment: survey, challenges, and future scope, IET Image Pro- cessing 13 (8) (2019) 1211–1224
2019
-
[20]
R. W. Strauss, T. R. Krieglstein, S. G. Priglinger, W. Reis, M. W. Ulbig, A. Kampik, A. S. Neubauer, Image quality characteristics of a novel colour scanning digital ophthalmoscope (sdo) com- pared with fundus photography, Ophthalmic and Physiological Optics 27 (6) (2007) 611–618
2007
-
[21]
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, arXiv preprint arXiv:2103.14030
-
[22]
Stergiou, R
A. Stergiou, R. Poppe, G. Kalliatakis, Refining activation down- sampling with softpool, in: Proceedings of the IEEE /CVF In- ternational Conference on Computer Vision, 2021, pp. 10357– 10366
2021
-
[23]
Girshick, Fast r-cnn, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp
R. Girshick, Fast r-cnn, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448. doi:10. 1109/ICCV.2015.169
2015
-
[24]
D. P. Kingma, J. Ba, Adam: A method for stochastic optimiza- tion, arXiv preprint arXiv:1412.6980
-
[25]
Mittal, A
A. Mittal, A. K. Moorthy, A. C. Bovik, No-reference image quality assessment in the spatial domain, IEEE Transactions on Image Processing 21 (12) (2012) 4695–4708. doi:10.1109/ TIP.2012.2214050
2012
-
[26]
Zhang, L
L. Zhang, L. Zhang, A. C. Bovik, A feature-enriched completely blind image quality evaluator, IEEE Transactions on Image Pro- cessing 24 (8) (2015) 2579–2591. doi:10.1109/TIP.2015. 2426416
2015 doi
-
[27]
A. K. Moorthy, A. C. Bovik, A two-step framework for con- structing blind image quality indices, IEEE Signal Processing Letters 17 (5) (2010) 513–516. doi:10.1109/LSP.2010. 2043888
2010 doi
-
[28]
A. K. Moorthy, A. C. Bovik, Blind image quality assessment: From natural scene statistics to perceptual quality, IEEE Trans- actions on Image Processing 20 (12) (2011) 3350–3364. doi: 10.1109/TIP.2011.2147325
2011
-
[29]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[30]
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)
-
[31]
J. Fang, L. Xie, X. Wang, X. Zhang, W. Liu, Q. Tian, Msg- transformer: Exchanging local spatial information by manipu- lating messenger tokens, in: CVPR, 2022
2022
-
[32]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. doi:10.1109/CVPR.2016.90. 11
2016 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.