REVIEW 4 major objections 4 minor 1 references
Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that an attribute-agnostic debiasing pipeline—unsupervised clustering plus gradient-similarity reweighting of a joint contrastive objective—reduces subgroup disparities in vision-language-model glaucoma screening without p
desk verdict Clever label-free debiasing recipe, but the fairness claim rests on an unvalidated cluster-demographic mapping. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a joint contrastive objective with per-cluster reweighting. The two losses—an image-text alignment loss and an image-pair contrastive loss—define learning signals for the same encoder; the cosine similarity between their gradients tells how much a cluster would be served by extra contrastive pressure. Clusters are found by unsupervised clustering of image embeddings, so no protected attributes are needed. The top-k weighting concentrates training on the clusters whose gradients are least aligned, which the paper identifies as the underperforming subgroups.
What would settle it
Take a retinal-image dataset with ground-truth demographic labels, run the clustering and reweighting pipeline, and compare the inferred clusters to the true groups. If the cluster assignments are nearly random with respect to demographics, or if Equalized Odds Distance computed with true labels does not improve even though cluster-based Equalized Odds Distance does, the central claim is falsified.
Extended reading notes
Core claim
The central claim is that demographic disparities in a vision-language model trained for glaucoma detection from retinal fundus images can be reduced without any protected attribute labels. The method (i) clusters image embeddings into proxy subgroups; (ii) for each cluster, computes gradient-similarity weights between the image-text alignment loss and the image-pair contrastive loss; and (iii) optimizes a joint, top-k weighted objective that upweights clusters whose gradients disagree most, targeting the hardest examples. On a public fairness benchmark for medical vision-language models, the paper reports lower Equalized Odds Distance, higher Equalized Subgroup AUC, and higher Groupwise AUC
Load-bearing premise
The method only removes bias if the unsupervised clusters of image embeddings are faithful stand-ins for the demographic subgroups that matter; if the clusters instead reflect imaging artifacts or disease severity, parity across clusters is not demographic fairness.
Editorial extensions
If this is right
- Glaucoma screening models built on vision-language models can be debiased without collecting sensitive demographic data, easing privacy and regulatory constraints.
- Fairness metrics can be computed and tracked during training even when protected attributes are absent, using the inferred clusters as stand-ins.
- The gradient-similarity, top-k reweighting recipe is not specific to glaucoma: it can be applied to any vision-language contrastive training pipeline.
- The reported gains (lower Equalized Odds Distance, higher subgroup AUC) would mean the worst-off inferred groups improve without requiring a separate debiasing stage.
Reading between the lines
- The central unvalidated premise is that clusters correspond to demographic groups; if they capture acquisition site, camera artifact, or disease severity, the fairness claim overstates real-world demographic parity. Testing on data with known demographic labels would resolve this.
- Because top-k selection is unsupervised, it may also emphasize clusters that are hard because of label noise or rare pathology; ablating top-k against uniform cluster weighting would show whether hard-cluster selection or simple reweighting drives the gain.
- The same weighting logic could be extended to other pairs of losses (e.g., supervised and self-supervised) as a general fairness-through-gradient-agreement tool.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an attribute-agnostic debiasing method for vision-language models (VLMs) in glaucoma screening from retinal fundus images. The pipeline is described in three steps: (i) infer 'proxy subgroups' by unsupervised clustering of image-image embeddings; (ii) compute gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss; and (iii) apply these weights in a joint, top-k weighted objective that upweights underperforming clusters. The method is evaluated on the Harvard FairVLMed glaucoma subset, with the abstract reporting reductions in Equalized Odds Distance (EOD) and gains in Equalized Subgroup AUC (ES AUC) and Groupwise AUC across inferred demographic subgroups.
Significance. If the central claim is correct, the method would address a real and important problem: mitigating demographic bias in medical imaging when protected attribute labels are unavailable. The idea of using gradient-similarity between multimodal and contrastive losses to guide reweighting, combined with cluster-based proxy subgroups, is technically interesting. However, the manuscript as submitted provides no quantitative results in the abstract, no code release, and the full text is corrupted to the point of unreadability. The empirical claims are therefore not currently falsifiable or reproducible. In addition, the evaluation is partially circular (fairness metrics are computed over the same clusters the method optimizes), and the central premise that clusters correspond to demographic subgroups is never validated. The significance of the work is conditional on resolving these issues with concrete evidence.
major comments (4)
- [Abstract and Full Text] The abstract states that the method 'reduces subgroup disparities' and lists EOD, ES AUC, and Groupwise AUC as evaluation metrics, but reports no numerical values. The provided full text is corrupted mojibake and includes an unrelated identifier 'arXiv:2508.09083v2 [physics.optics]', making it impossible to inspect any equations, tables, or experimental settings. Consequently, the central empirical claim cannot be verified. A clean, readable manuscript with quantitative results is essential.
- [Evaluation metrics (EOD, ES AUC, Groupwise AUC)] The fairness metrics are computed over the same inferred clusters that the top-k weighted objective (step iii) is designed to optimize. Because the reweighting explicitly upweights underperforming clusters, improving disparity metrics on that same partition is expected by construction. This circularity means the reported fairness gains do not establish demographic debiasing. The authors must also evaluate on ground-truth protected attributes (available in Harvard FairVLMed) and report cluster-to-attribute alignment (e.g., adjusted Rand index, normalized mutual information, or a cluster-by-attribute contingency table).
- [Step (i): proxy subgroups via clustering] The load-bearing assumption is that unsupervised clusters of image embeddings are faithful proxies for demographic subgroups. The abstract never validates this. If the clusters capture imaging device, acquisition site, disease severity, or retinal morphology, then parity across clusters does not constitute reduced demographic bias; it may even optimize a proxy that diverges from demographic parity. Concrete tests are needed: report cluster-by-attribute contingency, ARI/NMI against known attributes, and evaluate EOD and Groupwise AUC on the true demographic labels. Without this, the central fairness claim collapses.
- [Overall accuracy trade-off] The top-k reweighting may reduce group disparities at the expense of overall accuracy. The manuscript should report overall AUC or accuracy for the joint objective and compare it with the unweighted baseline, to show that any accuracy loss is acceptable. The abstract mentions only fairness metrics, and the corrupted full text does not allow checking this trade-off. This is a secondary but load-bearing issue for practical deployment.
minor comments (4)
- [Manuscript formatting] The full text is severely corrupted and unreadable; please resubmit a clean, properly encoded PDF or source file.
- [Inserted passage] The text contains an unrelated arXiv identifier 'arXiv:2508.09083v2 [physics.optics]' near the beginning; this appears to be a submission or rendering error and should be removed.
- [Terminology] The abstract uses 'inferred demographic subgroups' without explaining how clusters are interpreted as demographic. The authors should explicitly define the relationship between clusters and demographics, or use a more neutral term such as 'proxy subgroups' consistently.
- [Claim wording] The phrase 'without protected attribute supervision' is potentially misleading: the method does use unsupervised clustering of image embeddings, which may indirectly encode demographic information. Please clarify what information the clustering is allowed to use and what exactly is 'attribute-agnostic'.
Circularity Check
Reported fairness metrics are computed over the same inferred clusters that the top-k objective explicitly upweights, so the 'subgroup disparity reduction' is partly self-referential; demographic validity is not established.
-
self definitional
[Abstract, steps (i) and (iii) plus evaluation sentence]
"we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image-image embeddings, ... (iii) applies these weights in a joint, top-$k$ weighted objective to upweight underperforming clusters. ... reporting Equalized Odds Distance (EOD), Equalized Subgroup AUC (ES AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups."
The clusters from step (i) are the partition used in step (iii) to decide which clusters are 'underperforming' and are also the 'inferred demographic subgroups' over which EOD, ES AUC, and Groupwise AUC are measured. Because the loss explicitly upweights low-performing clusters, a reduction in these cluster-level disparity metrics is the optimization objective restated as an evaluation result, not an independent confirmation of demographic fairness. The demographic label is imported by naming, not verified: the abstract reports no validation of cluster-to-protected-attribute correspondence and no evaluation on the true demographic subgroups available in FairVLMed, so 'equitable performance across inferred demographic subgroups' is a self-referential claim about the method's own partition.
full rationale
The derivation chain in the abstract is: infer clusters from image-image embeddings, compute gradient-similarity weights, upweight underperforming clusters, then report EOD/ES AUC/Groupwise AUC across the inferred clusters. The evaluation partition is identical to the partition the method optimizes, so part of the measured fairness gain is coupled to the objective by construction. This is partial circularity: it does not force the central claim entirely, because the method could still fail to improve cluster parity and because true demographic debiasing is an extrapolation from cluster parity. The full text is corrupted in this version, so no equations or additional experimental details can be inspected; the circularity is apparent from the abstract alone. No self-citation or imported uniqueness theorem is involved. If the authors had validated cluster-demographic alignment (e.g., contingency tables or adjusted Rand index against FairVLMed protected attributes) or evaluated on observed demographic subgroups, the self-referential evaluation would be resolved; the abstract as written does not. Score 4 reflects this partial, evaluation-level circularity rather than a fully definitional equivalence.
Assumptions & free parameters
free parameters (3)
- number of clusters K (clustering resolution)
- top-k proportion (k in the top-k weighted objective)
- gradient-similarity weighting scale and normalization
assumptions (3)
- ad hoc to paper Unsupervised clusters of image embeddings are valid proxies for demographic subgroups.
- domain assumption Upweighting underperforming clusters reduces subgroup disparities without unacceptable overall accuracy loss.
- domain assumption Gradient similarity between the multimodal loss and the image-pair loss is a meaningful signal of subgroup difficulty.
invented entities (1)
-
proxy subgroups (inferred clusters treated as demographic subgroups)
Cite this review
Pith. "Pith review of Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision." pith.science (2026). https://pith.science/paper/BD37IPQ5
@misc{pith2026250809087,
author = {Pith},
title = {Pith review of: Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision},
year = {2026},
howpublished = {\url{https://pith.science/paper/BD37IPQ5}},
note = {Machine review of arXiv:2508.09087}
}
abstract
Vision-Language Models (VLMs) have achieved remarkable success on multimodal tasks such as image-text retrieval and zero-shot classification, yet they can exhibit demographic biases even when explicit protected attributes are absent during training. In this work, we focus on automated glaucoma screening from retinal fundus images, a critical application given that glaucoma is a leading cause of irreversible blindness and disproportionately affects underserved populations. Building on a reweighting-based contrastive learning framework, we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image-image embeddings, (ii) computes gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss, and (iii) applies these weights in a joint, top-$k$ weighted objective to upweight underperforming clusters. This label-free approach adaptively targets the hardest examples, thereby reducing subgroup disparities. We evaluate our method on the Harvard FairVLMed glaucoma subset, reporting Equalized Odds Distance (EOD), Equalized Subgroup AUC (ES AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups.
Reference graph
Works this paper leans on
-
[1]
����������� ������������������ ������ ������������� �������� �� ��� ��� ����� ��������� �� ������������� �� ������ � �� ����� �� ������ � �� �� ���������� � ��� �� ����� ��� � ���������� �� �������� ������ ��������� �� ����������� ����������������� ����� � ��������� ������� ������� ������ ��������� �� ����������� ����������������� ����� ������� ����� ��� ...
work page Pith review arXiv 2026
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.