Pith. sign in

REVIEW 4 major objections 3 minor 21 references

Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper tests whether removing sensitive-attribute information from retinal images improves fairness in diabetic retinopathy screening, and finds the outcome is model-dependent.

desk verdict Useful fairness benchmark on a new fundus dataset, but the disentanglement comparison conflates architecture and training changes with the disentanglement loss, so the headline model-dependent claim is not yet supported. read the letter →

arxiv 2507.09640 v1 pith:LO24FIYI submitted 2025-07-13 cs.CV cs.LG

classification cs.CVcs.LG
keywords diabeticretinopathyfundusimagingfairnessdisentanglementshortcutlearningmBRSETdeepmedical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a fairness technique called disentanglement can make deep learning models for diabetic retinopathy (DR) screening fairer without hurting accuracy, using the mBRSET dataset of handheld fundus images. It trains three architectures—ConvNeXt V2, DINOv2, and Swin V2—to predict DR and five sensitive attributes, then rebuilds each model with an autoencoder that tries to separate medical from sensitive information. The central finding is that this separation does not have a consistent benefit: it improves DINOv2's AUROC by 2 percentage points but lowers ConvNeXt V2 by 7 and Swin V2 by 3. The authors conclude that disentangling fine-grained retinal features is hard, and that fairness mitigation in this setting is non-trivial.

What carries the argument

The central mechanism is a disentangled autoencoder that maps a fundus image into two latent vectors, $E_{med}$ (intended to hold medical features) and $E_{sensit}$ (intended to hold sensitive-attribute features), alongside a DR classifier, a sensitive-attribute classifier, and a decoder. The disentanglement loss in Equation (1) adds Gaussian noise to one latent vector, regenerates the image, and penalizes changes in both vectors; the idea is that modifying one factor of variation should not alter the other. Classification and realism losses keep the two vectors useful and the reconstruction faithful. This machinery is what the paper varies across the three models to test whether removing SA-related information improves or harms DR prediction.

What would settle it

Measure the mutual information (or a dependence statistic) between $E_{med}$ and $E_{sensit}$ on the test set before and after disentanglement training; if the loss changes AUROC but leaves dependence near its baseline level, then the observed performance changes are not evidence about removing sensitive attributes.

Watch

Extended reading notes

Core claim

On the mBRSET fundus dataset, disentangling sensitive attributes from DR prediction produces model-dependent outcomes rather than a reliable fairness improvement. All three baselines predict DR well (up to 94% AUROC) and can predict age and sex with moderate to high accuracy, and fairness assessment reveals disparities such as DINOv2's 10% AUROC gap between age groups. Applying the proposed disentanglement architecture improved DINOv2's DR AUROC from 88% to 90%, but reduced ConvNeXt V2 from 94% to 87% and Swin V2 from 91% to 88%. The authors interpret this as evidence that some models rely on age- or insurance-related image features for DR prediction, while others lose useful medical detail when the targeted attribute is removed, and that latent dependencies between sensitive and medical features may persist after standard disentanglement.

Load-bearing premise

The load-bearing premise is that the disentanglement loss actually forces the medical and sensitive latent vectors to be statistically independent, so that changes in AUROC measure the effect of removing sensitive information rather than arbitrary information loss.

Editorial extensions

If this is right

  • If the central claim holds, model developers should not assume disentanglement will improve fairness in fundus imaging; the same mitigation can raise or lower AUROC by several points depending on the architecture.
  • A performance drop after removing age information (ConvNeXt V2: 94% to 87%) is evidence that some DR models rely on age-related image features, so this kind of intervention can diagnose shortcut dependence.
  • Fairness gaps in the baseline, such as DINOv2's 10% AUROC gap between age groups, mean pre-deployment fairness assessment should be groupwise, not only based on average performance.
  • Disentanglement can trade one disparity for another: for ConvNeXt V2 the age disparity fell from 4% to 1% while the insurance disparity rose from 3% to 21%.
  • Because latent dependencies may persist even after standard disentanglement, achieving true statistical independence between medical and sensitive features is a stronger goal than the tested loss provides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: A natural testable extension is to measure the mutual information between $E_{med}$ and $E_{sensit}$ before and after training; the paper's results would be mechanistically explained if reductions in this quantity, not changes in AUROC alone, predict fairer predictions.
  • Editorial inference: The model-dependence may partly reflect how strongly each architecture encodes the targeted attribute; architectures with strong SA prediction (such as ConvNeXt V2's 90.5% age AUROC) are the ones that lose performance when that information is removed.
  • Editorial inference: If disentanglement is applied per case, clinicians could use the risk distribution shifts (fewer false positives, more false negatives) to set decision thresholds, rather than treating AUROC as the sole success metric.
  • Editorial inference: The DINOv2 improvement came from disentangling sex, which the model predicted poorly (AUROC 53%), raising the question of whether disentangling an attribute the model barely encodes can still help fairness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This manuscript studies fairness and shortcut behavior of three deep models (ConvNeXt V2, DINOv2, Swin V2) for diabetic retinopathy (DR) prediction on the mBRSET retinal fundus dataset. It reports per-sensitive-attribute AUROC disparities, develops a disentangled autoencoder based on an existing architecture, and compares the disentangled models with the baselines for three selected cases (age for ConvNeXt V2, sex for DINOv2, insurance for Swin V2). The authors find that disentanglement improves DINOv2's AUROC by about 2% while lowering ConvNeXt V2 and Swin V2 by about 7% and 3%, and conclude that disentanglement is not a reliably beneficial fairness mitigation technique in this domain.

Significance. If the headline conclusions were supported, the result would be a useful cautionary empirical data point for fairness mitigation in medical imaging, since most prior ophthalmology disentanglement work targets anonymization or domain generalization rather than fairness. The descriptive baseline results on a recent handheld-camera dataset, including subgroup AUROC disparities, decision curve analysis, and risk distribution plots, are a useful contribution. I also note the absence of circularity: the disentanglement architecture is taken from prior work and the loss weights are fixed before evaluation, so the benchmark numbers are not fitted to the desired conclusions. The main weakness is that the comparison supporting the headline claim is not an ablation of the disentanglement term.

major comments (4)
  1. [§3.2 vs. §3.4, Eq. (4)] The disentanglement condition is confounded with a complete architecture change. Baselines are pretrained classifiers fine-tuned with focal loss (hidden dim 128, learning rate 1e-5, batch size 2-4), whereas the disentangled model is a custom autoencoder with a 256-dim bottleneck, decoder, reconstruction losses, an SA classifier, and different optimizer settings (learning rate 5e-5, batch size 32, weight decay 1e-6). Consequently, the 94->87, 88->90, and 91->88 AUROC changes in Section 4.2 cannot be attributed to Ldisent; they may reflect the encoder capacity, the reconstruction objective, or the training recipe. The central claim that the effect is model-dependent requires a control that toggles only the disentanglement loss, e.g., the same autoencoder trained with and without λd Ldisent, or the same pretrained backbone with an added disentanglement regularizer.
  2. [§4.2, first paragraph] The three reported disentanglement cases are not a systematic comparison. Each model is paired with a different sensitive attribute (age, sex, insurance), and the pairs are selected after observing the largest baseline disparities and SA predictability. Because the selection rule depends on the outcomes being studied, the observed pattern (one gain, two drops) could be an artifact of which attribute was chosen for each model. The paper should either report the full 3x5 matrix of disentanglement runs, or a prespecified subset, or explicitly treat the current results as exploratory case studies rather than evidence about model dependence.
  3. [§4.1, Table 2; §4.2, Fig. 2] All AUROC numbers are single-split point estimates. In Table 2 several subgroups contain only 39 or 40 observations (insurance group 1, obesity group 1), and the abstract-level comparison relies on differences of 2-7 AUROC points (88->90 vs 94->87 vs 91->88). Without confidence intervals, bootstrap resampling, or multiple seeds, these differences are within plausible sampling noise, especially for the smallest subgroups. Please provide uncertainty estimates or additional splits, and avoid interpreting 2-3 point changes as meaningful without them.
  4. [§3.4, Eq. (1); §4.2, Summary] The disentanglement loss in Eq. (1) penalizes changes in Emed and Esensit when Gaussian noise is added to one latent vector, but this does not by itself guarantee statistical independence of the two encodings. The paper's own Section 4.2 acknowledges that latent dependencies may persist after standard disentanglement [7]. Unless independence or non-predictability of the SA from Emed is measured (e.g., by SA prediction AUROC from Emed alone or mutual information estimates), the observed AUROC changes cannot be interpreted as the effect of removing sensitive information; they may reflect arbitrary information loss. At minimum, report the SA-prediction performance of the disentangled Emed.
minor comments (3)
  1. [§1, footnote 5] The GitHub link is 'https://github.com/public_if_accepted', which is not a real repository; either provide the actual anonymized repository or revise the claim that code is publicly available.
  2. [Equation (1)] The notation in Eq. (1) is under-specified: Ii is introduced as 'a new, altered image' but the conditioning of the expectation and the precise dependence of Ii on the perturbed latent are not formalized; please define Emed, Esensit, Iori, and Ii explicitly.
  3. [Table 2] Table 2 reports 100.00% AUROC for the obesity subgroup for all three models; with N=40 this warrants a comment on the subgroup's class balance and the stability of this estimate, otherwise the reader may over-interpret a degenerate subgroup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports external empirical benchmarks on mBRSET, with hyperparameters fixed before evaluation and the borrowed architecture used as a method source, not as evidence for the measured outcomes.

full rationale

The central claims are empirical: baseline and disentangled models are trained on the external mBRSET dataset, and the reported AUROC values, disparities, and DCA/risk-distribution comparisons are measured results rather than quantities implied by construction. The disentanglement architecture is adapted from reference [9], which shares a co-author (J. S. Cardoso), but this self-citation supplies only the architectural template and training losses; it does not supply the AUROC numbers, fairness gaps, or the conclusion that disentanglement effects are model-dependent. No parameter in the loss was fitted to reproduce the target AUROC values; lambda_r = 1 and lambda_d = 5 are stated as fixed choices before evaluation. The three representative cases in Section 4.2 were selected by observed disparities and SA predictability, which is a selection and confound concern, not circularity: the reported differences still come from held-out test evaluation. The paper also openly flags the limitation that latent dependencies may persist after disentanglement (Section 4.2, citing [7]), which undermines the interpretation of the loss but does not constitute a self-referential reduction. There is no self-definitional step, no fitted-input-called-prediction, no uniqueness theorem imported from the authors, and no renaming of a known result as an organizing contribution. Therefore the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are postulated; the latent vectors are internal representations of a known architecture. The main dependencies are hand-set loss weights, a non-quantified image-quality filter, and the unproven independence assumption behind Eq. 1.

free parameters (4)
  • lambda_d = 5
    Weight on the disentanglement loss in Eq. 4, chosen by hand. It controls the strength of the independence penalty and therefore affects all disentangled results.
  • lambda_r = 1
    Weight on the realism loss in Eq. 4, chosen by hand. It balances reconstruction quality against the other objectives.
  • alpha = 48
    PSNR normalization threshold in the realism loss, Eq. 3. It is a hand-set constant that affects the realism loss scale.
  • latent_dimension = 256
    Dimensionality of the Emed and Esensit latent vectors in the disentanglement autoencoder, chosen without reported ablation. It determines how much information the latent split can preserve.
assumptions (5)
  • domain assumption DR severity binarization using ICDR levels 0-1 as Normal and 2-4 as Referable
    Used in Section 3.1. Assumes the clinical cutoff is meaningful for the fairness and performance analysis.
  • domain assumption Only high-quality, macula-centered images are used
    Section 3.1. The filtering criteria are not quantified, and differential exclusion could bias the sample toward easier or harder images.
  • ad hoc to paper Gaussian noise perturbation of one latent vector forces independence of Emed and Esensit
    Section 3.4, Eq. 1. The loss penalizes changes in both latent vectors after perturbation, but no proof or measurement guarantees statistical independence.
  • domain assumption Pretrained ImageNet features transfer to fundus images
    Section 3.2. All models are pretrained and normalized using ImageNet statistics; transferability to retinal images is assumed.
  • domain assumption Group fairness can be assessed with AUROC disparities
    Sections 3.3 and 4.1. The analysis uses AUROC gaps, DCA, and risk distributions without statistical tests or confidence intervals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams." pith.science (2026). https://pith.science/paper/LO24FIYI

@misc{pith2026250709640,
  author       = {Pith},
  title        = {Pith review of: Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LO24FIYI}},
  note         = {Machine review of arXiv:2507.09640}
}
read the original abstract

Diabetic retinopathy (DR) is a leading cause of vision loss in working-age adults. While screening reduces the risk of blindness, traditional imaging is often costly and inaccessible. Artificial intelligence (AI) algorithms present a scalable diagnostic solution, but concerns regarding fairness and generalization persist. This work evaluates the fairness and performance of image-trained models in DR prediction, as well as the impact of disentanglement as a bias mitigation technique, using the diverse mBRSET fundus dataset. Three models, ConvNeXt V2, DINOv2, and Swin V2, were trained on macula images to predict DR and sensitive attributes (SAs) (e.g., age and gender/sex). Fairness was assessed between subgroups of SAs, and disentanglement was applied to reduce bias. All models achieved high DR prediction performance in diagnosing (up to 94% AUROC) and could reasonably predict age and gender/sex (91% and 77% AUROC, respectively). Fairness assessment suggests disparities, such as a 10% AUROC gap between age groups in DINOv2. Disentangling SAs from DR prediction had varying results, depending on the model selected. Disentanglement improved DINOv2 performance (2% AUROC gain), but led to performance drops in ConvNeXt V2 and Swin V2 (7% and 3%, respectively). These findings highlight the complexity of disentangling fine-grained features in fundus imaging and emphasize the importance of fairness in medical imaging AI to ensure equitable and reliable healthcare solutions.

Figures

Figures reproduced from arXiv: 2507.09640 by the authors.

Figure 1
Figure 1. Overview of the disentanglement network. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of baseline and disentangled model performance for Con [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [7]

    Medical Image Analysis 102, 103529 (May 2025)

    Liu, X., Li, B., Vernooij, M.W., Wolvius, E.B., Roshchupkin, G.V., Bron, E.E.: Ai-based association analysis for medical imaging using latent-space geomet- ric confounder correction. Medical Image Analysis 102, 103529 (May 2025). https://doi.org/10.1016/j.media.2025.103529

  2. [9]

    Medical Image Analysis 95, 103209 (Jul 2024)

    Montenegro, H., Cardoso, J.S.: Anonymizing medical case-based explanations through disentanglement. Medical Image Analysis 95, 103209 (Jul 2024). https://doi.org/10.1016/j.media.2024.103209, https://linkinghub.elsevier. com/retrieve/pii/S1361841524001348

  3. [1]

    https://doi.org/10.48550/ARXIV.2102.06764, https://arxiv.org/abs/2102

    Cherepanova, V., Nanda, V., Goldblum, M., Dickerson, J.P., Gold- stein, T.: Technical challenges for training fair neural networks (2021). https://doi.org/10.48550/ARXIV.2102.06764, https://arxiv.org/abs/2102. 06764, arXiv preprint arXiv:2102.06764

  4. [2]

    Generalizing Across Domains in Diabetic Retinopathy via Variational Autoencoders

    Chokuwa, S., Khan, M.H.: Generalizing across domains in diabetic retinopathy via variational autoencoders (2023). https://doi.org/10.48550/ARXIV.2309.11301, https://arxiv.org/abs/2309.11301

  5. [3]

    In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Com- puter Vision and Pattern Recognition. p. 248–255. IEEE, Miami, FL (Jun 2009). https://doi.org/10.1109/CVPR.2009.5206848, https://ieeexplore.ieee. org/document/5206848/

  6. [4]

    13804, p

    Du, S., Hers, B., Bayasi, N., Hamarneh, G., Garbi, R.: FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning, vol. 13804, p. 185–202. Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031- 25069-9_13, https://link.springer.com/10.1007/978-3-031-25069-9_13

  7. [5]

    https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014). https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980

  8. [6]

    https://doi.org/10.48550/ARXIV.1708.02002, https://arxiv

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection (2017). https://doi.org/10.48550/ARXIV.1708.02002, https://arxiv. org/abs/1708.02002

Show all 21 references
  1. [8]

    https://doi.org/10.48550/ARXIV.2103.14030, https://arxiv.org/abs/ 2103.14030

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows (2021). https://doi.org/10.48550/ARXIV.2103.14030, https://arxiv.org/abs/ 2103.14030

  2. [10]

    In- dian Journal of Endocrinology and Metabolism 26(2), 111–118 (Mar 2022)

    Mounirou, B.A.M., Adam, N.D., Yakoura, A.K.H., Aminou, M.S.M., Liu, Y.T., Tan, L.Y.: Diabetic retinopathy: An overview of treatments. In- dian Journal of Endocrinology and Metabolism 26(2), 111–118 (Mar 2022). https://doi.org/10.4103/ijem.ijem_480_21, https://journals.lww.com/...

  3. [11]

    https://doi.org/10.48550/ARXIV.2402.19186, https://arxiv.org/abs/2402

    Müller, S., Koch, L.M., Lensch, H.P.A., Berens, P.: Disentangling representations of retinal images with generative models (2024). https://doi.org/10.48550/ARXIV.2402.19186, https://arxiv.org/abs/2402. 19186

  4. [12]

    https://doi.org/10.13026/QXPD- 1Y65, https://physionet.org/content/mbrset/1.0/

    Nakayama, L.F., Zago Ribeiro, L., Restrepo, D., Santos Barboza, N., Dias Fiter- man,R.,VieiraSousa,M.l.,Pereira,A.D.A.,Regatieri,C.,Malerbi,F.K.,Andrade, R.: mbrset, a mobile brazilian retinal dataset. https://doi.org/10.13026/QXPD- 1Y65, https://physionet.org/content/mbrset/1.0/

  5. [13]

    Fernandes et al

    Ong Ly, C., Unnikrishnan, B., Tadic, T., Patel, T., Duhamel, J., Kan- del, S., Moayedi, Y., Brudno, M., Hope, A., Ross, H., McIntosh, C.: Short- cut learning in medical ai hinders generalization: method for estimating ai 10 L. Fernandes et al. model generalization without exte...

  6. [14]

    https://doi.org/10.48550/ARXIV.2304.07193, https://arxiv.org/ abs/2304.07193

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J....

  7. [15]

    12729, p

    Ouyang, J., Adeli, E., Pohl, K.M., Zhao, Q., Zaharchuk, G.: Representation Disen- tanglement for Multi-modal Brain MRI Analysis, vol. 12729, p. 321–333. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-78191- 0_25, https://link.springer.com/10....

  8. [16]

    PLOS ONE 17(10), e0274098 (Oct 2022)

    Trivedi, A., Robinson, C., Blazes, M., Ortiz, A., Desbiens, J., Gupta, S., Dodhia, R., Bhatraju, P.K., Liles, W.C., Kalpathy-Cramer, J., Lee, A.Y., Lavista Ferres, J.M.: Deep learning models for covid-19 chest x-ray classification: Preventing short- cut learning using feature ...

  9. [17]

    https://doi.org/10.48550/ARXIV.2301.00808, https://arxiv.org/abs/ 2301.00808

    Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., Xie, S.: Convnext v2: Co-designing and scaling convnets with masked autoencoders (2023). https://doi.org/10.48550/ARXIV.2301.00808, https://arxiv.org/abs/ 2301.00808

  10. [18]

    In: Linguraru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A

    Xia, P., Hu, M., Tang, F., Li, W., Zheng, W., Ju, L., Duan, P., Yao, H., Ge, Z.: Generalizing to unseen domains in diabetic retinopathy with disentangled repre- sentations. In: Linguraru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A. (eds....

  11. [19]

    Xu, Z., Li, J., Yao, Q., Li, H., Zhou, S.K.: Fairness in medi- cal image analysis and healthcare: A literature survey (oct 2023). https://doi.org/10.36227/techrxiv.24324979, https://www.techrxiv.org/ articles/preprint/Fairness_in_Medical_Image_Analysis_and_Healthcare_ A_Litera...

  12. [20]

    Nature Medicine 30(10), 2838–2848(Oct2024).https://doi.org/10.1038/s41591-024-03113-4, https://www

    Yang, Y., Zhang, H., Gichoya, J.W., Katabi, D., Ghassemi, M.: The limits of fair medical imaging ai in real-world generalization. Nature Medicine 30(10), 2838–2848(Oct2024).https://doi.org/10.1038/s41591-024-03113-4, https://www. nature.com/articles/s41591-024-03113-4

  13. [21]

    Biomedical Optics Express14(10), 5466 (Oct 2023)

    Zhao, Z., Faghihroohi, S., Yang, J., Huang, K., Navab, N., Maier, M., Nasseri, M.A.: Unobtrusive biometric data de-identification of fundus images using la- tent space disentanglement. Biomedical Optics Express14(10), 5466 (Oct 2023). https://doi.org/10.1364/BOE.495438, https:...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.