REVIEW 4 major objections 3 minor 21 references
Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams
T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper tests whether removing sensitive-attribute information from retinal images improves fairness in diabetic retinopathy screening, and finds the outcome is model-dependent.
desk verdict Useful fairness benchmark on a new fundus dataset, but the disentanglement comparison conflates architecture and training changes with the disentanglement loss, so the headline model-dependent claim is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a disentangled autoencoder that maps a fundus image into two latent vectors, $E_{med}$ (intended to hold medical features) and $E_{sensit}$ (intended to hold sensitive-attribute features), alongside a DR classifier, a sensitive-attribute classifier, and a decoder. The disentanglement loss in Equation (1) adds Gaussian noise to one latent vector, regenerates the image, and penalizes changes in both vectors; the idea is that modifying one factor of variation should not alter the other. Classification and realism losses keep the two vectors useful and the reconstruction faithful. This machinery is what the paper varies across the three models to test whether removing SA-related information improves or harms DR prediction.
What would settle it
Measure the mutual information (or a dependence statistic) between $E_{med}$ and $E_{sensit}$ on the test set before and after disentanglement training; if the loss changes AUROC but leaves dependence near its baseline level, then the observed performance changes are not evidence about removing sensitive attributes.
Extended reading notes
Core claim
On the mBRSET fundus dataset, disentangling sensitive attributes from DR prediction produces model-dependent outcomes rather than a reliable fairness improvement. All three baselines predict DR well (up to 94% AUROC) and can predict age and sex with moderate to high accuracy, and fairness assessment reveals disparities such as DINOv2's 10% AUROC gap between age groups. Applying the proposed disentanglement architecture improved DINOv2's DR AUROC from 88% to 90%, but reduced ConvNeXt V2 from 94% to 87% and Swin V2 from 91% to 88%. The authors interpret this as evidence that some models rely on age- or insurance-related image features for DR prediction, while others lose useful medical detail when the targeted attribute is removed, and that latent dependencies between sensitive and medical features may persist after standard disentanglement.
Load-bearing premise
The load-bearing premise is that the disentanglement loss actually forces the medical and sensitive latent vectors to be statistically independent, so that changes in AUROC measure the effect of removing sensitive information rather than arbitrary information loss.
Editorial extensions
If this is right
- If the central claim holds, model developers should not assume disentanglement will improve fairness in fundus imaging; the same mitigation can raise or lower AUROC by several points depending on the architecture.
- A performance drop after removing age information (ConvNeXt V2: 94% to 87%) is evidence that some DR models rely on age-related image features, so this kind of intervention can diagnose shortcut dependence.
- Fairness gaps in the baseline, such as DINOv2's 10% AUROC gap between age groups, mean pre-deployment fairness assessment should be groupwise, not only based on average performance.
- Disentanglement can trade one disparity for another: for ConvNeXt V2 the age disparity fell from 4% to 1% while the insurance disparity rose from 3% to 21%.
- Because latent dependencies may persist even after standard disentanglement, achieving true statistical independence between medical and sensitive features is a stronger goal than the tested loss provides.
Reading between the lines
- Editorial inference: A natural testable extension is to measure the mutual information between $E_{med}$ and $E_{sensit}$ before and after training; the paper's results would be mechanistically explained if reductions in this quantity, not changes in AUROC alone, predict fairer predictions.
- Editorial inference: The model-dependence may partly reflect how strongly each architecture encodes the targeted attribute; architectures with strong SA prediction (such as ConvNeXt V2's 90.5% age AUROC) are the ones that lose performance when that information is removed.
- Editorial inference: If disentanglement is applied per case, clinicians could use the risk distribution shifts (fewer false positives, more false negatives) to set decision thresholds, rather than treating AUROC as the sole success metric.
- Editorial inference: The DINOv2 improvement came from disentangling sex, which the model predicted poorly (AUROC 53%), raising the question of whether disentangling an attribute the model barely encodes can still help fairness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript studies fairness and shortcut behavior of three deep models (ConvNeXt V2, DINOv2, Swin V2) for diabetic retinopathy (DR) prediction on the mBRSET retinal fundus dataset. It reports per-sensitive-attribute AUROC disparities, develops a disentangled autoencoder based on an existing architecture, and compares the disentangled models with the baselines for three selected cases (age for ConvNeXt V2, sex for DINOv2, insurance for Swin V2). The authors find that disentanglement improves DINOv2's AUROC by about 2% while lowering ConvNeXt V2 and Swin V2 by about 7% and 3%, and conclude that disentanglement is not a reliably beneficial fairness mitigation technique in this domain.
Significance. If the headline conclusions were supported, the result would be a useful cautionary empirical data point for fairness mitigation in medical imaging, since most prior ophthalmology disentanglement work targets anonymization or domain generalization rather than fairness. The descriptive baseline results on a recent handheld-camera dataset, including subgroup AUROC disparities, decision curve analysis, and risk distribution plots, are a useful contribution. I also note the absence of circularity: the disentanglement architecture is taken from prior work and the loss weights are fixed before evaluation, so the benchmark numbers are not fitted to the desired conclusions. The main weakness is that the comparison supporting the headline claim is not an ablation of the disentanglement term.
major comments (4)
- [§3.2 vs. §3.4, Eq. (4)] The disentanglement condition is confounded with a complete architecture change. Baselines are pretrained classifiers fine-tuned with focal loss (hidden dim 128, learning rate 1e-5, batch size 2-4), whereas the disentangled model is a custom autoencoder with a 256-dim bottleneck, decoder, reconstruction losses, an SA classifier, and different optimizer settings (learning rate 5e-5, batch size 32, weight decay 1e-6). Consequently, the 94->87, 88->90, and 91->88 AUROC changes in Section 4.2 cannot be attributed to Ldisent; they may reflect the encoder capacity, the reconstruction objective, or the training recipe. The central claim that the effect is model-dependent requires a control that toggles only the disentanglement loss, e.g., the same autoencoder trained with and without λd Ldisent, or the same pretrained backbone with an added disentanglement regularizer.
- [§4.2, first paragraph] The three reported disentanglement cases are not a systematic comparison. Each model is paired with a different sensitive attribute (age, sex, insurance), and the pairs are selected after observing the largest baseline disparities and SA predictability. Because the selection rule depends on the outcomes being studied, the observed pattern (one gain, two drops) could be an artifact of which attribute was chosen for each model. The paper should either report the full 3x5 matrix of disentanglement runs, or a prespecified subset, or explicitly treat the current results as exploratory case studies rather than evidence about model dependence.
- [§4.1, Table 2; §4.2, Fig. 2] All AUROC numbers are single-split point estimates. In Table 2 several subgroups contain only 39 or 40 observations (insurance group 1, obesity group 1), and the abstract-level comparison relies on differences of 2-7 AUROC points (88->90 vs 94->87 vs 91->88). Without confidence intervals, bootstrap resampling, or multiple seeds, these differences are within plausible sampling noise, especially for the smallest subgroups. Please provide uncertainty estimates or additional splits, and avoid interpreting 2-3 point changes as meaningful without them.
- [§3.4, Eq. (1); §4.2, Summary] The disentanglement loss in Eq. (1) penalizes changes in Emed and Esensit when Gaussian noise is added to one latent vector, but this does not by itself guarantee statistical independence of the two encodings. The paper's own Section 4.2 acknowledges that latent dependencies may persist after standard disentanglement [7]. Unless independence or non-predictability of the SA from Emed is measured (e.g., by SA prediction AUROC from Emed alone or mutual information estimates), the observed AUROC changes cannot be interpreted as the effect of removing sensitive information; they may reflect arbitrary information loss. At minimum, report the SA-prediction performance of the disentangled Emed.
minor comments (3)
- [§1, footnote 5] The GitHub link is 'https://github.com/public_if_accepted', which is not a real repository; either provide the actual anonymized repository or revise the claim that code is publicly available.
- [Equation (1)] The notation in Eq. (1) is under-specified: Ii is introduced as 'a new, altered image' but the conditioning of the expectation and the precise dependence of Ii on the perturbed latent are not formalized; please define Emed, Esensit, Iori, and Ii explicitly.
- [Table 2] Table 2 reports 100.00% AUROC for the obesity subgroup for all three models; with N=40 this warrants a comment on the subgroup's class balance and the stability of this estimate, otherwise the reader may over-interpret a degenerate subgroup.
Circularity Check
No significant circularity: the paper reports external empirical benchmarks on mBRSET, with hyperparameters fixed before evaluation and the borrowed architecture used as a method source, not as evidence for the measured outcomes.
full rationale
The central claims are empirical: baseline and disentangled models are trained on the external mBRSET dataset, and the reported AUROC values, disparities, and DCA/risk-distribution comparisons are measured results rather than quantities implied by construction. The disentanglement architecture is adapted from reference [9], which shares a co-author (J. S. Cardoso), but this self-citation supplies only the architectural template and training losses; it does not supply the AUROC numbers, fairness gaps, or the conclusion that disentanglement effects are model-dependent. No parameter in the loss was fitted to reproduce the target AUROC values; lambda_r = 1 and lambda_d = 5 are stated as fixed choices before evaluation. The three representative cases in Section 4.2 were selected by observed disparities and SA predictability, which is a selection and confound concern, not circularity: the reported differences still come from held-out test evaluation. The paper also openly flags the limitation that latent dependencies may persist after disentanglement (Section 4.2, citing [7]), which undermines the interpretation of the loss but does not constitute a self-referential reduction. There is no self-definitional step, no fitted-input-called-prediction, no uniqueness theorem imported from the authors, and no renaming of a known result as an organizing contribution. Therefore the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- lambda_d =
5
- lambda_r =
1
- alpha =
48
- latent_dimension =
256
assumptions (5)
- domain assumption DR severity binarization using ICDR levels 0-1 as Normal and 2-4 as Referable
- domain assumption Only high-quality, macula-centered images are used
- ad hoc to paper Gaussian noise perturbation of one latent vector forces independence of Emed and Esensit
- domain assumption Pretrained ImageNet features transfer to fundus images
- domain assumption Group fairness can be assessed with AUROC disparities
Cite this review
Pith. "Pith review of Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams." pith.science (2026). https://pith.science/paper/LO24FIYI
@misc{pith2026250709640,
author = {Pith},
title = {Pith review of: Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams},
year = {2026},
howpublished = {\url{https://pith.science/paper/LO24FIYI}},
note = {Machine review of arXiv:2507.09640}
}
read the original abstract
Diabetic retinopathy (DR) is a leading cause of vision loss in working-age adults. While screening reduces the risk of blindness, traditional imaging is often costly and inaccessible. Artificial intelligence (AI) algorithms present a scalable diagnostic solution, but concerns regarding fairness and generalization persist. This work evaluates the fairness and performance of image-trained models in DR prediction, as well as the impact of disentanglement as a bias mitigation technique, using the diverse mBRSET fundus dataset. Three models, ConvNeXt V2, DINOv2, and Swin V2, were trained on macula images to predict DR and sensitive attributes (SAs) (e.g., age and gender/sex). Fairness was assessed between subgroups of SAs, and disentanglement was applied to reduce bias. All models achieved high DR prediction performance in diagnosing (up to 94% AUROC) and could reasonably predict age and gender/sex (91% and 77% AUROC, respectively). Fairness assessment suggests disparities, such as a 10% AUROC gap between age groups in DINOv2. Disentangling SAs from DR prediction had varying results, depending on the model selected. Disentanglement improved DINOv2 performance (2% AUROC gain), but led to performance drops in ConvNeXt V2 and Swin V2 (7% and 3%, respectively). These findings highlight the complexity of disentangling fine-grained features in fundus imaging and emphasize the importance of fairness in medical imaging AI to ensure equitable and reliable healthcare solutions.
Figures
Reference graph
Works this paper leans on
-
[7]
Medical Image Analysis 102, 103529 (May 2025)
Liu, X., Li, B., Vernooij, M.W., Wolvius, E.B., Roshchupkin, G.V., Bron, E.E.: Ai-based association analysis for medical imaging using latent-space geomet- ric confounder correction. Medical Image Analysis 102, 103529 (May 2025). https://doi.org/10.1016/j.media.2025.103529
arXiv 2025
-
[9]
Medical Image Analysis 95, 103209 (Jul 2024)
Montenegro, H., Cardoso, J.S.: Anonymizing medical case-based explanations through disentanglement. Medical Image Analysis 95, 103209 (Jul 2024). https://doi.org/10.1016/j.media.2024.103209, https://linkinghub.elsevier. com/retrieve/pii/S1361841524001348
-
[1]
https://doi.org/10.48550/ARXIV.2102.06764, https://arxiv.org/abs/2102
Cherepanova, V., Nanda, V., Goldblum, M., Dickerson, J.P., Gold- stein, T.: Technical challenges for training fair neural networks (2021). https://doi.org/10.48550/ARXIV.2102.06764, https://arxiv.org/abs/2102. 06764, arXiv preprint arXiv:2102.06764
-
[2]
Generalizing Across Domains in Diabetic Retinopathy via Variational Autoencoders
Chokuwa, S., Khan, M.H.: Generalizing across domains in diabetic retinopathy via variational autoencoders (2023). https://doi.org/10.48550/ARXIV.2309.11301, https://arxiv.org/abs/2309.11301
work page Pith review arXiv doi:10.48550/arxiv.2309.11301 2023
-
[3]
In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition. p. 248–255. IEEE, Miami, FL (Jun 2009). https://doi.org/10.1109/CVPR.2009.5206848, https://ieeexplore.ieee. org/document/5206848/
arXiv 2009
-
[4]
Du, S., Hers, B., Bayasi, N., Hamarneh, G., Garbi, R.: FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning, vol. 13804, p. 185–202. Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031- 25069-9_13, https://link.springer.com/10.1007/978-3-031-25069-9_13
-
[5]
https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014). https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980
-
[6]
https://doi.org/10.48550/ARXIV.1708.02002, https://arxiv
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection (2017). https://doi.org/10.48550/ARXIV.1708.02002, https://arxiv. org/abs/1708.02002
Show all 21 references
-
[8]
https://doi.org/10.48550/ARXIV.2103.14030, https://arxiv.org/abs/ 2103.14030
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows (2021). https://doi.org/10.48550/ARXIV.2103.14030, https://arxiv.org/abs/ 2103.14030
-
[10]
In- dian Journal of Endocrinology and Metabolism 26(2), 111–118 (Mar 2022)
Mounirou, B.A.M., Adam, N.D., Yakoura, A.K.H., Aminou, M.S.M., Liu, Y.T., Tan, L.Y.: Diabetic retinopathy: An overview of treatments. In- dian Journal of Endocrinology and Metabolism 26(2), 111–118 (Mar 2022). https://doi.org/10.4103/ijem.ijem_480_21, https://journals.lww.com/...
2022 doi
- [11]
-
[12]
https://doi.org/10.13026/QXPD- 1Y65, https://physionet.org/content/mbrset/1.0/
Nakayama, L.F., Zago Ribeiro, L., Restrepo, D., Santos Barboza, N., Dias Fiter- man,R.,VieiraSousa,M.l.,Pereira,A.D.A.,Regatieri,C.,Malerbi,F.K.,Andrade, R.: mbrset, a mobile brazilian retinal dataset. https://doi.org/10.13026/QXPD- 1Y65, https://physionet.org/content/mbrset/1.0/
-
[13]
Fernandes et al
Ong Ly, C., Unnikrishnan, B., Tadic, T., Patel, T., Duhamel, J., Kan- del, S., Moayedi, Y., Brudno, M., Hope, A., Ross, H., McIntosh, C.: Short- cut learning in medical ai hinders generalization: method for estimating ai 10 L. Fernandes et al. model generalization without exte...
2024 doi
-
[14]
https://doi.org/10.48550/ARXIV.2304.07193, https://arxiv.org/ abs/2304.07193
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J....
-
[15]
12729, p
Ouyang, J., Adeli, E., Pohl, K.M., Zhao, Q., Zaharchuk, G.: Representation Disen- tanglement for Multi-modal Brain MRI Analysis, vol. 12729, p. 321–333. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-78191- 0_25, https://link.springer.com/10....
2021 doi
-
[16]
PLOS ONE 17(10), e0274098 (Oct 2022)
Trivedi, A., Robinson, C., Blazes, M., Ortiz, A., Desbiens, J., Gupta, S., Dodhia, R., Bhatraju, P.K., Liles, W.C., Kalpathy-Cramer, J., Lee, A.Y., Lavista Ferres, J.M.: Deep learning models for covid-19 chest x-ray classification: Preventing short- cut learning using feature ...
2022 doi
-
[17]
https://doi.org/10.48550/ARXIV.2301.00808, https://arxiv.org/abs/ 2301.00808
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., Xie, S.: Convnext v2: Co-designing and scaling convnets with masked autoencoders (2023). https://doi.org/10.48550/ARXIV.2301.00808, https://arxiv.org/abs/ 2301.00808
-
[18]
In: Linguraru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A
Xia, P., Hu, M., Tang, F., Li, W., Zheng, W., Ju, L., Duan, P., Yao, H., Ge, Z.: Generalizing to unseen domains in diabetic retinopathy with disentangled repre- sentations. In: Linguraru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A. (eds....
2024 doi
-
[19]
Xu, Z., Li, J., Yao, Q., Li, H., Zhou, S.K.: Fairness in medi- cal image analysis and healthcare: A literature survey (oct 2023). https://doi.org/10.36227/techrxiv.24324979, https://www.techrxiv.org/ articles/preprint/Fairness_in_Medical_Image_Analysis_and_Healthcare_ A_Litera...
2023
-
[20]
Nature Medicine 30(10), 2838–2848(Oct2024).https://doi.org/10.1038/s41591-024-03113-4, https://www
Yang, Y., Zhang, H., Gichoya, J.W., Katabi, D., Ghassemi, M.: The limits of fair medical imaging ai in real-world generalization. Nature Medicine 30(10), 2838–2848(Oct2024).https://doi.org/10.1038/s41591-024-03113-4, https://www. nature.com/articles/s41591-024-03113-4
-
[21]
Biomedical Optics Express14(10), 5466 (Oct 2023)
Zhao, Z., Faghihroohi, S., Yang, J., Huang, K., Navab, N., Maier, M., Nasseri, M.A.: Unobtrusive biometric data de-identification of fundus images using la- tent space disentanglement. Biomedical Optics Express14(10), 5466 (Oct 2023). https://doi.org/10.1364/BOE.495438, https:...
2023 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.