{"id":"95c76839-4551-442b-834d-c7f55f66e98e","arxiv_id":"2411.12273","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new fundus image quality dataset with continuous scores and a Transformer-hypernetwork regressor that claims, but does not consistently report, state-of-the-art performance.","lead":"The authors built a new dataset of 2,246 retinal fundus photos with continuous quality scores from six ophthalmologists, plus a Transformer-based hypernetwork model that predicts those scores. The model reports strong correlation with human scores, but the paper's headline numbers conflict between the abstract and its own tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal inconsistency in reported results: abstract numbers (PLCC 0.9423, SRCC 0.9488) and §3.2 improvement claims do not match Table 1; no error bars support 'significantly outperforming' (§3.1), so the central claim is unsupported by the paper's own evidence.","rationale":"Central claim: FTHNet predicts MOS with PLCC 0.9423 and SRCC 0.9488 on FQS and significantly outperforms prior IQA methods with fewer parameters. For the claim to hold, the reported numbers must be reproducible from the paper's own evidence and 'significantly' must be statistically supported. The least secure condition is the reported evidence itself: the abstract and §3.2 text disagree with Table 1, and the model configurations are described inconsistently (§2.2.1 vs §3.3.3). The abstract's '10-fold cross-validation' is actually a repeated random 80/15/5 split (§3.1) with no per-fold results or error bars, so 'significantly outperforming' is unsupported; from Table 1 the SRCC margin over HyperIQA is 0.0007. The reader noted the headline/table mismatch in the rationale but named split leakage as the weakest assumption; I agree on the verdict while locating the load-bearing problem in the self-contradictory reported numbers (partial agreement). The leakage concern remains a valid secondary threat: §2.1.1 selects 2246 images from over 10,000 eye instances but never states whether multiple images from one patient or eye were grouped within a split, so if the numeric inconsistencies were ever repaired, the evaluation protocol would still need checking. Secondary flag: Eq. (1) describes λ1=0.11 as the junior weight and λ2=0.22 as the experienced weight, yet multiplies the experienced scores ΣOi by λ1 and the junior scores ΣOji by λ2, contradicting its own caption and giving juniors twice the influence on the ground-truth MOS. Credit where due: the dataset contribution, the documented labeling process with SD statistics (§2.1.3), honest failed-case reporting (§4.4), and a real deployment sketch all have value, but none verifies the headline numbers, and the dataset/code release is only promised (§5). Because the mismatch is checkable immediately from the manuscript, it is the most decisive concern; the concrete test settles it. Verdict unchanged: REJECT.","tokens_in":14226,"tokens_out":16149,"duration_ms":137594,"concrete_test":"Run an internal consistency audit on the manuscript's own numbers: (a) recompute the §3.2 improvement deltas from Table 1 (PLCC: 0.9442-0.9355=0.0087 vs claimed 0.0133; SRCC: 0.9358-0.9351=0.0007 vs claimed 0.0072); (b) search Tables 1-3 for any configuration yielding the abstract's SRCC 0.9488 and PLCC 0.9423; (c) check that the FTHNet-S/L definitions in §2.2.1 match the winning rows of Table 3. If no configuration reproduces the abstract numbers, request the authors' per-round 80/15/5 test results with means and standard deviations; the headline claim stands only if an identified run yields the abstract's SRCC and the margin over HyperIQA exceeds one standard deviation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's performance statement: 'FTHNet can give quality scores ... with PLCC of 0.9423 and SRCC of 0.9488, significantly outperforming other methods with fewer parameters and less computation complexity.' For this claim to hold, the reported numbers must be internally consistent and the comparison statistically grounded. Neither condition is met as written.\n\n(1) The abstract's numbers appear nowhere in the results. Table 1 reports FTHNet-S SRCC 0.9330/PLCC 0.9435 and FTHNet-L SRCC 0.9358/PLCC 0.9442; no row in Tables 1-3 contains SRCC 0.9488 or PLCC 0.9423. (2) The §3.2 improvement claims do not match Table 1: the text credits FTHNet-L with PLCC gains of 0.0133/0.0183/0.5161/0.6690 over GraphIQA/HyperIQA/TRIQ/DeepIQA, but Table 1 yields 0.0087/0.0137/0.5115/0.6644; the SRCC gains (0.0143/0.0072/0.5270/0.6459) likewise differ from computed values (0.0078/0.0007/0.5205/0.6394). (3) The model identity is unstable: §2.2.1 defines FTHNet-L as (2,4,6,2)/C64, but §3.3.3/Table 3 identify the best configuration as (2,2,6,2)/C64, and Table 1's FTHNet-S (0.9330/0.9435) differs from Table 3's identical (2,4,6,2)/C32 row (0.9303/0.9424). (4) The claimed statistical foundation is absent: the abstract says '10-fold cross-validation to ensure the significance of the results', but §3.1 describes a repeated random 80/15/5 split for 10 rounds, not 10-fold CV, and no per-round results or error bars are reported. From Table 1, FTHNet-L's SRCC margin over HyperIQA is 0.0007, so 'significantly outperforming' is unsubstantiated. The dataset and code are only promised (§5), and §4.4 honestly documents large errors on Reject images but does not repair the headline claim. Because the manuscript's own evidence is self-contradictory, the central claim as stated cannot be verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a fundus image quality assessment (FIQA) dataset, FQS, containing 2246 fundus images with continuous MOS and three-level quality labels, and proposes FTHNet, a Transformer-based hypernetwork that predicts quality scores. The authors report that FTHNet outperforms several existing IQA methods with fewer parameters and less computation, and they provide deployment experiments and inference time measurements. Evaluation is performed on the authors' own dataset with random 80/15/5 splits repeated ten times.","tokens_in":14719,"tokens_out":6149,"duration_ms":49936,"significance":"If the reported performance were reliable, the FQS dataset and the lightweight FTHNet architecture would be a useful contribution to FIQA, where continuous quality scores are still uncommon and real-time deployment is clinically relevant. The authors are to be credited for constructing a clinical dataset with a documented labeling protocol and for reporting deployment latency. However, the central claims are not currently substantiated: the abstract's headline numbers do not appear in the results tables, the improvement claims in the text do not match Table 1, and the evaluation protocol is described inconsistently. As written, the significance cannot be assessed beyond the potential value of the dataset itself.","major_comments":[{"comment":"The abstract reports PLCC 0.9423 and SRCC 0.9488 for FTHNet, but these values do not appear anywhere in Tables 1, 2, or 3. Table 1 reports FTHNet-S with PLCC 0.9435 and SRCC 0.9330, and FTHNet-L with PLCC 0.9442 and SRCC 0.9358. Since the central performance claim in the abstract is not reproducible from the paper's own reported results, the headline result is unsupported.","section":"Abstract; Tables 1-3"},{"comment":"The text states that FTHNet-L achieves PLCC improvements of 0.0133, 0.0183, 0.5161, and 0.6690 over GraphIQA, HyperIQA, TRIQ, and DeepIQA, respectively. Computing the same differences from Table 1 gives 0.0087, 0.0137, 0.5115, and 0.6644. The claimed SRCC improvements (0.0143, 0.0072, 0.5270, 0.6459) also differ from the table-derived values (0.0078, 0.0007, 0.5205, 0.6394). The reported advantage is therefore not backed by the tabulated data.","section":"Section 3.2 vs Table 1"},{"comment":"The identity of the proposed models is inconsistent. Section 2.2.1 defines FTHNet-L as (2,4,6,2) with C=64, but Section 3.3.3 states that FTHNet-L is (2,2,6,2) with C=64, and Table 3 lists both configurations with different results. Moreover, Table 1's FTHNet-S (SRCC 0.9330, PLCC 0.9435) differs from Table 3's row for the identical configuration (2,4,6,2)/C32 (SRCC 0.9303, PLCC 0.9424). It is unclear which configuration actually produced the reported results.","section":"Section 2.2.1, Section 3.3.3, Tables 1 and 3"},{"comment":"The abstract claims that all experiments used 10-fold cross-validation, but Section 3.1 describes a random 80/15/5 split repeated for 10 rounds, which is not 10-fold cross-validation. No per-round results or error bars are reported, so the assertion of statistical significance in Section 3.2 is not supported. In addition, Section 2.1.1 and Section 3.1 do not state whether multiple images from the same eye or patient were grouped into the same split, leaving open the possibility of patient-level leakage that could inflate the reported correlations.","section":"Abstract vs Section 3.1; Sections 2.1.1 and 3.2"}],"minor_comments":[{"comment":"The text says lambda1 and lambda2 represent the weights of junior and experienced ophthalmologists, but the equation multiplies Oi (the experienced scores) by lambda1 and Oji (the junior scores) by lambda2, so the assignment appears reversed and should be corrected.","section":"Eq. (1) and following text"},{"comment":"The caption contains a typo: 'Exapmples' should be 'Examples'.","section":"Figure 1 caption"},{"comment":"The loss ablation includes an 'L1+L2' row that is not described in the text, and the formatting of 'L smoothL1' is inconsistent with the other rows.","section":"Table 2"},{"comment":"The text reports 5.67 M parameters for FTHNet-S, while Table 1 lists 5.662 M and Table 3 lists 5.558 M for the configuration described as FTHNet-S; these numbers should be reconciled.","section":"Section 3.2 vs Table 1"},{"comment":"The phrase 'significantly outperforming' is used without any significance test or confidence interval; please either provide statistical evidence or use weaker wording.","section":"Section 3.2"},{"comment":"The deployment experiment is anecdotal and does not quantify clinical utility, for example by measuring agreement between the system's quality scores and clinicians' decisions on a separate validation set.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The internal inconsistencies are pervasive and affect the central claims: the abstract's performance numbers do not match any table, the text's improvements do not match Table 1, and the model configuration is described differently across sections. These are not merely presentation issues; they undermine the reproducibility of the reported results. If the authors can provide a consistent set of final numbers, per-round results, and a clear split protocol, a resubmission could be considered, but as written the manuscript does not support its main conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper has a serious reporting problem: the abstract's headline numbers (PLCC 0.9423, SRCC 0.9488) do not appear in Table 1. FTHNet-S gives 0.9435/0.9330, FTHNet-L gives 0.9442/0.9358. The improvement claims in §3.2 also don't match the table; the PLCC margins are off by roughly 0.0046 across all four baselines, and the SRCC margins are off by varying amounts. This isn't a typo—the text and table appear to describe different experiments.\n\nWhat's genuinely good: the FQS dataset. If released, 2246 fundus images with continuous MOS and three-level labels is a useful asset. The labeling process is thoughtful—reference set, six ophthalmologists, documented weighting, and an SD distribution that supports consistency. The deployment experiment is a concrete plus. The architecture is incremental (Swin-style backbone plus hypernetwork), but the combination is reasonable and the ablation study is fairly thorough.\n\nThe soft spots are load-bearing for the central claim. The '10-fold cross-validation' is actually repeated random 80/15/5 splits over 10 rounds, which isn't k-fold CV, and no per-round results or error bars are provided. So 'significantly outperforming' is unsupported; FTHNet-L's SRCC edge over HyperIQA is 0.0007 per Table 1. The model identity is unstable: FTHNet-L is defined as (2,4,6,2)/C64 in §2.2.1, but §3.3.3 says the best is (2,2,6,2)/C64, and FTHNet-S in Table 1 doesn't match the same configuration in Table 3. The dataset and code are only promised, so there's no way to verify the results. The fail-case discussion in §4.4 is honest but doesn't address the headline.\n\nFor whom? A researcher working on FIQA could get value from the dataset and the deployment details, but the paper in its current form can't be used as a reliable benchmark because the internal numbers contradict each other.\n\nI'd send this to peer review—the dataset is potentially valuable and the problems are fixable with a careful revision—but my own verdict would be reject as written. A referee should demand reconciliation of abstract/table numbers, per-round results or confidence intervals, and a concrete release plan for data and code.","headline":"FQS dataset is a genuinely useful resource in principle, but the paper's own tables contradict its headline numbers and the statistical claim is unsupported as written.","tokens_in":15367,"tokens_out":4358,"would_cite":false,"duration_ms":37671,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Model scores fundus image quality at 0.94 correlation","keywords":["fundus image quality assessment","mean opinion score","transformer","hypernetwork","regression","ophthalmology","clinical dataset","image quality assessment"],"falsifier":"Re-run FTHNet's evaluation with the folds regrouped so that every image of the same patient or eye lies in a single training or test partition; if SRCC or PLCC falls substantially below the reported roughly 0.94, the claimed generalization is inflated by patient-level leakage.","tokens_in":14002,"feed_emoji":"👁️","tokens_out":10334,"duration_ms":100303,"temperature":0.7,"pith_summary":"Fundus image quality is usually judged in coarse categories, which makes it hard to compare images within a category or to set precise clinical thresholds. This paper tries to replace that with a continuous quality score, and to make the score practical enough for real-time use. To do that it introduces the Fundus Quality Score dataset of 2,246 clinical fundus images labeled with a 0-100 mean opinion score and a three-level category, and FTHNet, a transformer-hypernetwork that regresses the score; the evaluation uses repeated 80/15/5 random splits across ten rounds. The paper reports correlations of 0.9423 (PLCC) and 0.9488 (SRCC) with six ophthalmologists' ratings on its dataset, and shows the model running fast enough for deployment in an automatic diagnosis pipeline. If the result holds, eye-care teams could reject or recapture bad images at the camera and compare image quality objectively across patients and devices.","feed_headline":"Model scores fundus image quality at 0.94 correlation","feed_subtitle":"A lightweight transformer-hypernetwork and a 2,246-image clinical dataset give eye doctors a continuous, comparable quality score.","key_machinery":"The load-bearing object is FTHNet, a four-part architecture: a shifted-window transformer backbone that extracts multi-resolution features; a Distortion Perception Network that condenses each resolution into a compact vector; a Parameter Hypernetwork that reads the deepest feature map and generates the weight and bias parameters; and a five-layer Target Network that maps the condensed distortion vectors to a score in [0,100]. The hypernetwork does the central work: because it produces the regressor's parameters from the input image itself, the scoring function can adapt to the particular mix of blur, haze, uneven illumination, and over-darkness in each fundus image. The paper's ablation supports this role: replacing the hypernetwork with directly learned parameters drops SRCC from 0.9358 to 0.6092. The shifted-window attention in the backbone keeps computational cost linear in spatial size, which is how the model stays light enough for the reported sub-60 ms inference times.","core_discovery":"The paper's central claim is that fundus image quality can be predicted as a continuous score by a lightweight network that combines a shifted-window transformer backbone with a hypernetwork that generates the weights of a small regression head from the image's own features. The FQS dataset supplies the ground truth: each of 2,246 fundus images carries a mean opinion score between 0 and 100 obtained as a weighted average of six ophthalmologists' ratings, plus a Good/Usable/Reject label. On this dataset the paper reports Pearson correlation 0.9423 and Spearman correlation 0.9488 between predicted and human-assigned scores, saying FTHNet outperforms the compared model-based and deep-learning IQA methods while using fewer parameters and less computation. The authors also report that ablation results show removing the hypernetwork from FTHNet drops SRCC from 0.9358 to 0.6092, and that deployed as an API the model scores a single fundus image in about 45-56 ms.","pith_inferences":["A test the paper leaves implicit is cross-device generalization: retraining on its two capture devices and testing on fundus images from other cameras would show whether the 0.94 correlation is tied to specific equipment or reflects a device-invariant notion of clinical quality.","The labeling rubric explicitly weights visibility of the macula, optic disc, and vessel structure, so FTHNet's score may be a better proxy for diagnostic usability than generic IQA measures; one could test that by measuring whether a downstream disease detector's errors concentrate in images FTHNet scores low.","The FQS annotation protocol, including a reference set, weighted senior and junior ratings, and standard-deviation monitoring, could be transferred to build continuous quality labels for other imaging modalities such as OCT, where similar clinical-quality needs exist.","If the hypernetwork genuinely adapts to distortion type, feeding the same image with different synthetic distortions should move the generated target-network parameters in clusterable, distortion-specific ways; checking that would give an interpretable handle on what drives the score."],"forward_implications":["If the reported correlations hold outside the FQS splits, a continuous quality score gives hospitals an objective threshold for whether a fundus image is acceptable for diagnosis, instead of relying on broad category labels.","The small model sizes (5.66M and 14.88M parameters) and sub-60 ms per-image inference would allow quality checking to run during acquisition, so clinicians can immediately re-shoot unreadable images.","Because FTHNet's regression head is generated per image by the hypernetwork, the same architecture could be adapted to quality scoring of other clinical image types with diverse distortions.","The FQS dataset, with continuous MOS and three-level labels from multiple ophthalmologists, provides a common test bed for other FIQA algorithms to be retrained and compared on the same scale."],"supporting_citations":[{"why":"Introduces the self-adaptive hypernetwork concept that FTHNet adapts for fundus image quality regression.","marker":"[13]"},{"why":"Supplies the shifted-window multi-head self-attention building block that keeps FTHNet's backbone linear in spatial size.","marker":"[21]"},{"why":"State-of-the-art blind IQA baseline that FTHNet compares against in the main results table.","marker":"[14]"},{"why":"Transformer-based IQA baseline used for comparison and one of the methods FTHNet outperforms.","marker":"[15]"},{"why":"Deep IQA baseline used in the comparison study, representing CNN-style quality regression.","marker":"[12]"},{"why":"Clinical study cited for the claim that a substantial fraction of fundus images are unreadable, motivating the FIQA task.","marker":"[8]"},{"why":"Defines SoftPool, the downsampling operation used in the Distortion Perception Network and bias generation.","marker":"[22]"},{"why":"Model-based no-reference IQA baseline retrained and compared on FQS.","marker":"[25]"}],"fun_headline_variants":["Fundus IQ: 0.94 correlation with doctors' scores","New net rates retina images at 0.94 agreement","Lightweight hypernetwork scores eye images at 0.94","0.94 correlation: AI rates fundus quality in 50ms","Continuous retina quality score from lightweight AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random 80/15/5 splits keep all images from the same patient and the same eye in a single partition, a condition the paper does not explicitly confirm; if violated, the reported held-out correlations would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Fundus IQ: 0.94 correlation with doctors' scores","New net rates retina images at 0.94 agreement","Lightweight hypernetwork scores eye images at 0.94","0.94 correlation: AI rates fundus quality in 50ms","Continuous retina quality score from lightweight AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2689,"prompt_tokens":999,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1607}},"tokens_in":615,"tokens_out":1690,"duration_ms":15842,"temperature":1.0,"reasoning_tokens":1607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:43:52.033536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run FTHNet's evaluation with the folds regrouped so that every image of the same patient or eye lies in a single training or test partition; if SRCC or PLCC falls substantially below the reported roughly 0.94, the claimed generalization is inflated by patient-level leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the self-adaptive hypernetwork concept that FTHNet adapts for fundus image quality regression."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"State-of-the-art blind IQA baseline that FTHNet compares against in the main results table."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Transformer-based IQA baseline used for comparison and one of the methods FTHNet outperforms."},{"cited_title":"Philip, L","cited_arxiv_id":null,"evidence_quote":"Clinical study cited for the claim that a substantial fraction of fundus images are unreadable, motivating the FIQA task."},{"cited_title":"Stergiou, R","cited_arxiv_id":null,"evidence_quote":"Defines SoftPool, the downsampling operation used in the Distortion Perception Network and bias generation."}],"review_version":1}