REVIEW 5 major objections 5 minor 28 references
Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift
T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read DRiFt splits medical image and text features into clinical and spurious components, and this explicit decoupling lifts few-shot accuracy and robustness under distribution shift.
desk verdict A plausible few-shot medical VLM adaptation recipe whose core decoupling mechanism is underspecified and whose robustness claim rests on a single seed; worth refereeing, but only with heavy revisions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The feature-decoupling step is the machinery. A projection function phi_M acts on each raw embedding z_M (M for vision or text) and splits it into z_{M,u} (invariant, clinically relevant) and z_{M,s} (spurious). The three losses are then targeted: the contrastive loss aligns only invariant embeddings across modalities; the KL-divergence loss drives predictions from spurious embeddings toward a uniform class distribution so they carry no classification signal; and conditional independence losses reduce the mutual information between invariant and spurious embedding parts given the label. This separation is what lets the model learn clinical cues without being misled by domain-specific artifac
What would settle it
Take the same pipeline but replace the projection functions with a fixed random linear map that does not separate clinical content. If average Top-1 accuracy remains near 45.8%, the decoupling losses are not doing the claimed work; if it falls sharply, the projection choice is the critical ingredient. Alternatively, rerun the 16-shot setting with ten different random seeds and check whether the +11.4 average advantage over the baseline persists.
Extended reading notes
Core claim
The central claim is that spurious associations, such as a clothing shadow or incidental report wording, can be suppressed if each embedding is decomposed into invariant and spurious parts and the training losses act on those parts separately. DRiFt projects image and text embeddings into two subspaces, aligns only the invariant subspace with a contrastive loss, pushes the spurious subspace's predictions toward a uniform distribution, and penalizes conditional dependence between the two subspaces given the label. Trained with 16 samples per class, low-rank adapters, and learnable prompts, the framework reports an average Top-1 accuracy of 45.8% across eight medical tasks, compared with 34.4%
Load-bearing premise
The load-bearing premise is that the fixed projection functions used to split embeddings into invariant and spurious parts are meaningful; if those projections are arbitrary or poorly matched to the data, the three losses cannot actually enforce decoupling and the reported gains would come from another mechanism.
Editorial extensions
If this is right
- If DRiFt's claim holds, then few-shot adaptation of medical vision-language models can be done without full fine-tuning, since only low-rank adapters and prompt tokens are updated.
- Explicit spurious-feature neutralization can be added to any prompt-based method as a loss term, potentially improving out-of-distribution behavior.
- The decoupling works on both modalities, so degradation from text-side spurious cues (e.g., incidental findings in reports) is also addressed.
- Cross-dataset transfer suggests that models trained on one imaging protocol can preserve accuracy on another, which matters for multi-site deployment.
- Using two prompt tokens rather than one nearly doubled average accuracy in the experiments, so prompt capacity itself appears to be part of the mechanism.
Reading between the lines
- Because the projection functions that split embeddings are not trained and their architecture is not specified, the decoupling effect may be sensitive to how those projections are chosen; a direct test is to make them learnable or compare several fixed choices.
- The few-shot numbers come from a single 16-sample-per-class split with no seeds or error bars, so the +11.4 point gap might not survive resampling; reporting multiple splits would separate mechanism from sampling luck.
- The caption-generation step relies on an instruction-tuned captioning model; if caption quality degrades for rare diseases, the invariant alignment could reinforce wrong text-image associations, making caption validation a hidden dependency.
- The same loss design could transfer to other high-stakes domains with known spurious cues, such as pathology or retinal OCT, where domain shift follows similar patterns, but this extrapolation is not tested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. DRiFt is a parameter-efficient few-shot adaptation framework for medical vision-language models under distribution shift. It decomposes image and text embeddings into invariant (clinically relevant) and spurious components via unspecified projection functions, fine-tunes only LoRA adapters and prompt tokens, and trains with a combination of invariant contrastive alignment, KL-based spurious neutralization, and conditional-independence (HSIC) losses. The authors add InstructBLIP-generated captions to MedIMeta and evaluate on eight in-distribution tasks and cross-dataset transfer against MaPLe-IC and CoOp-OOD-IC. The abstract reports +11.4% Top-1 and +3.3% Macro-F1 over prior prompt-based methods while maintaining robustness across unseen datasets.
Significance. The intended contribution—explicit cross-modal decoupling of clinical and class-agnostic features with parameter-efficient tuning—is timely and, if valid, would be useful for few-shot medical VLM adaptation. The paper has concrete strengths: code is released, two prompt-based baselines are adapted to the image-caption setting, and ablations cover shots, prompt depth, prompt tokens, and backbone. However, the central decoupling mechanism is not actually specified, the HSIC estimator and loss weights are absent, and the reported in-distribution AUC is at chance for most tasks. As a result, the paper's core claims—both the mechanism and the robustness of the gains—are not currently established.
major comments (5)
- [Section 2.1] The decomposition is undefined. The paper states only z_{M i,u}=phi_M(zMi) and calls these 'two projection functions'; it neither defines the spurious component z_{M i,s}, nor specifies the architecture of phi_M (linear/MLP/random), nor includes phi_M in the trainable set Theta=theta∪gamma. If phi_M is frozen, the invariant/spurious split is fixed before training and Eqs. (1)-(3) cannot adaptively enforce decoupling; if phi_M is learned, it is omitted from the implementation. Either way, the advertised cause of the +11.4% gain is unverifiable.
- [Section 2.2] The conditional-independence losses L_conv and L_cont in Eq. (2) are written as HSIC terms, but no estimator, kernel, or conditioning procedure is given. The paper does not report how I(z_v;z_s|Y) is computed in practice, so Eq. (2) and the overall loss L in Eq. (3) are not reproducible. This is a load-bearing component of the claimed decoupling.
- [Section 2.2 and 3.1] The loss weights alpha and beta in Eq. (3) are never reported, and the implementation details do not describe a validation split for choosing them, the LoRA rank r, or the caption-generation thresholds. Ablations vary prompt depth and token count without indicating where selection occurs, raising a risk of test-set-based selection. Reproducibility and the interpretation of the reported gains are therefore limited.
- [Table 1] AUC is at or below chance for most tasks (bus 34.9, skinl_derm 49.5, derm 46.5, glaucoma 51.2, pneumonia 52.4, mammo_mass 45.9; average 49.1 with chance 50), directly contradicting the abstract's 'strong robustness' and 'reliable' claims. In addition, all results come from a single 16-sample-per-class draw with no error bars or significance tests. With few-shot sample sizes, the Top-1 improvements may be within run-to-run noise.
- [Table 2 and Section 3.2] The cross-dataset generalization claim is not supported. DRiFt is best only for skinl_derm (10.4 vs 6.4-6.5) and is worse on derm (17.8 vs 18.8-18.9) while roughly tying on glaucoma and fundus. No average or statistical comparison is reported, so 'maintaining strong robustness across unseen datasets' is unsubstantiated. Moreover, the method extends prior work [9], but [9] is not among the baselines, leaving the incremental contribution unclear.
minor comments (5)
- [Section 3.1] Typo: 'howvere' should be 'however'. Also 'spuriousrelationshipswhichcanbeverysensitive' is missing spaces.
- [Section 2.2] Notation is inconsistent: Eq. (2) defines L_conv and L_cont, but Eq. (3) introduces L_con = avg(L_conv, L_cont); the names should be unified.
- [Section 2.2] The reference to 'Figure 4(left)' appears in Section 2.2, before the experiments are described; this should be moved to the results discussion.
- [Table 1 / Section 3.2] The pneumonia row for DRiFt (37.5% Acc, 18.2% Macro-F1) is identical to CoOp-OOD-IC, yet the text says 'pneumonia accuracy improves'; the comparison baseline should be stated precisely.
- [Section 4a] The shot-ablation text says bus 'increases' from 50% to 42.3% (decrease); the sentence is internally contradictory and should be corrected.
Circularity Check
Load-bearing decoupling premise is self-cited to the authors' Dimple preprint; no equation-level circularity in the reported accuracy gains.
-
self citation load bearing
[Section 2.1, 'Decoupled Feature Representation']
"To enhance generalization, we decompose each image and text embedding into invariant and spurious components [9]. Given an input embeddingzMi, two projection functions are defined byz Mi,u =ϕ M (zMi), whereM=v,t, withvandtrepresenting visual and textual embeddings, respectively. This decomposition ensures clinically meaningful features are preserved while mitigating domain-specific artifacts."
The paper's central mechanism—explicit decoupling into invariant and spurious components—is introduced by citing [9], the authors' own prior arXiv preprint (Dimple), rather than by deriving or validating the projection functions. The projection functions phi_M are not defined, parameterized, or included in the trainable set Θ=θ∪γ, so the claimed guarantee that the decomposition preserves clinical features and mitigates artifacts rests entirely on the self-cited prior work. This is load-bearing: if [9] is not accepted, the losses L_ce^u, L_sp^r, and L_con in Section 2.2 have no specified spurious component to act on, so the mechanism behind the +11.4% improvement is not independently established in this paper.
full rationale
The central accuracy claims in Table 1 are arithmetic differences against external baselines (MaPLe-IC and CoOp-OOD-IC), so the headline +11.4%/+3.3% is not circular. No equation-level reduction of a prediction to a fitted parameter was found; the losses in Eq. (3) are not equivalent to the reported metrics by construction. However, the mechanism said to produce the gain—the invariant/spurious split—is imported from the authors' prior work [9] (Dimple) and the projection functions phi_M are never specified or included in the trainable set Θ=θ∪γ. The paper's only justification for 'This decomposition ensures clinically meaningful features are preserved' is the self-citation to [9], which is an unreviewed preprint by the same authors. This makes the decoupling premise load-bearing self-citation rather than independent derivation. The underspecification of phi_M is also a reproducibility gap, but it is not itself circular; it only reinforces that the claimed mechanism rests on [9]. Hyperparameter choices (n=2, J=3) are reported and ablated on test data, a mild selection feedback, but they are not presented as predictions derived from the model. Overall: some self-citation load-bearing; the main empirical comparison is independent.
Assumptions & free parameters
free parameters (6)
- alpha (spurious neutralization weight)
- beta (independence loss weight)
- Number of prompt tokens =
2
- Prompt depth (J) =
3
- LoRA rank r
- Caption generation minimum image threshold =
100+
assumptions (5)
- domain assumption The pre-trained ViT-B/16-based vision-language model provides sufficient transferable visual and textual representations.
- domain assumption InstructBLIP-generated captions, after manual filtering, are clinically relevant and provide the supervision signal for decoupling.
- ad hoc to paper A fixed projection phi_M can separate invariant from spurious features without being trained or verified.
- domain assumption HSIC-based conditional independence measures enforce the intended invariance in both modalities.
- domain assumption The MedIMeta labels and task splits are reliable and representative.
Cite this review
Pith. "Pith review of Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift." pith.science (2026). https://pith.science/paper/2ME5JMIU
@misc{pith2026250909397,
author = {Pith},
title = {Pith review of: Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift},
year = {2026},
howpublished = {\url{https://pith.science/paper/2ME5JMIU}},
note = {Machine review of arXiv:2509.09397}
}
read the original abstract
Medical vision-language models (VLMs) offer promise for clinical decision support, yet their reliability under distribution shifts remains a major concern for safe deployment. These models often learn task-agnostic correlations due to variability in imaging protocols and free-text reports, limiting their generalizability and increasing the risk of failure in real-world settings. We propose DRiFt, a structured feature decoupling framework that explicitly separates clinically relevant signals from task-agnostic noise using parameter-efficient tuning (LoRA) and learnable prompt tokens. To enhance cross-modal alignment and reduce uncertainty, we curate high-quality, clinically grounded image-text pairs by generating captions for a diverse medical dataset. Our approach improves in-distribution performance by +11.4% Top-1 accuracy and +3.3% Macro-F1 over prior prompt-based methods, while maintaining strong robustness across unseen datasets. Ablation studies reveal that disentangling task-relevant features and careful alignment significantly enhance model generalization and reduce unpredictable behavior under domain shift. These insights contribute toward building safer, more trustworthy VLMs for clinical use. The code is available at https://github.com/rumaima/DRiFt.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[9]
Umaima Rahman, Mohammad Yaqub, and Dwarikanath Mahapatra. Dimple– disentangled multi-modal prompt learning: Enhancing out-of-distribution align- ment with invariant and spurious feature separation.arXiv preprint arXiv:2506.21237, 2025
arXiv 2025
-
[1]
Which clinical decisions benefit from automation? a task complexity approach.International journal of medical infor- matics, 70(2-3):309–316, 2003
Vitali Sintchenko and Enrico W Coiera. Which clinical decisions benefit from automation? a task complexity approach.International journal of medical infor- matics, 70(2-3):309–316, 2003
2003
-
[2]
Mrishift: Disentangled repre- sentation learning for 3d mri lesion segmentation under distributional shifts
Umaima Rahman, Guangyi Chen, and Kun Zhang. Mrishift: Disentangled repre- sentation learning for 3d mri lesion segmentation under distributional shifts. In 2024 12th European Workshop on Visual Information Processing (EUVIP), pages 1–6. IEEE, 2024
2024
-
[3]
Heterogeneity in systematic reviews of medical imaging diagnos- tic test accuracy studies: a systematic review.JAMA Network Open, 7(2):e240649– e240649, 2024
Samuel J White, Qi Sheng Phua, Lucy Lu, Kaspar L Yaxley, Matthew DF Mcinnes, and Minh-Son To. Heterogeneity in systematic reviews of medical imaging diagnos- tic test accuracy studies: a systematic review.JAMA Network Open, 7(2):e240649– e240649, 2024
2024
-
[4]
Reducing reliance on spurious features in medical image clas- sification with spatial specificity
Khaled Saab, Sarah Hooper, Mayee Chen, Michael Zhang, Daniel Rubin, and Christopher Ré. Reducing reliance on spurious features in medical image clas- sification with spatial specificity. InMachine Learning for Healthcare Conference, pages 760–784. PMLR, 2022
2022
-
[5]
When more is less: Incorporating additional datasets can hurt performance by introducing spuri- ous correlations
Rhys Compton, Lily Zhang, Aahlad Puli, and Rajesh Ranganath. When more is less: Incorporating additional datasets can hurt performance by introducing spuri- ous correlations. InMachine Learning for Healthcare Conference, pages 110–127. PMLR, 2023
2023
-
[6]
Maple: Multi-modal prompt learning
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113–19122, 2023
2023
-
[7]
Conditional prompt learning for vision-language models
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022
2022
Show all 28 references
-
[8]
Umaima Rahman, Raza Imam, Mohammad Yaqub, Boulbaba Ben Amor, and Dwarikanath Mahapatra. Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts? In2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), page...
2025
-
[10]
Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022
2022
-
[11]
Test- time low rank adaptation via confidence maximization for zero-shot generalization of vision-language models.arXiv preprint arXiv:2407.15913, 2024
Raza Imam, Hanan Gani, Muhammad Huzaifa, and Karthik Nandakumar. Test- time low rank adaptation via confidence maximization for zero-shot generalization of vision-language models.arXiv preprint arXiv:2407.15913, 2024
2024 arXiv
-
[12]
A compre- hensive and easy-to-use multi-domain multi-task medical imaging meta-dataset (medimeta).arXiv preprint arXiv:2404.16000, 2024
Stefano Woerner, Arthur Jaques, and Christian F Baumgartner. A compre- hensive and easy-to-use multi-domain multi-task medical imaging meta-dataset (medimeta).arXiv preprint arXiv:2404.16000, 2024. 10 Rahman et al
2024
-
[13]
Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023
2023
-
[14]
Domain generalization on medical imaging classification us- ing episodic training with task augmentation.Computers in biology and medicine, 141:105144, 2022
Chenxin Li, Xin Lin, Yijin Mao, Wei Lin, Qi Qi, Xinghao Ding, Yue Huang, Dong Liang, and Yizhou Yu. Domain generalization on medical imaging classification us- ing episodic training with task augmentation.Computers in biology and medicine, 141:105144, 2022
2022
-
[15]
Domain generalization for medical image analysis: A review.Proceedings of the IEEE, 2024
Jee Seok Yoon, Kwanseok Oh, Yooseung Shin, Maciej A Mazurowski, and Heung-Il Suk. Domain generalization for medical image analysis: A review.Proceedings of the IEEE, 2024
2024
-
[16]
Evaluation of do- main generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine.Scientific reports, 12(1):2726, 2022
Lin Lawrence Guo, Stephen R Pfohl, Jason Fries, Alistair EW Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung. Evaluation of do- main generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine.Scientific r...
2022
-
[17]
Few-shot adaptation of medical vision-language models
Fereshteh Shakeri, Yunshi Huang, Julio Silva-Rodríguez, Houda Bahig, An Tang, Jose Dolz, and Ismail Ben Ayed. Few-shot adaptation of medical vision-language models. InInternational Conference on Medical Image Computing and Computer- Assisted Intervention, pages 553–563. Springer, 2024
2024
-
[18]
Clipath: Fine-tune clip with visual feature fusion for pathology image analysis towards minimizing data collection efforts
Zhengfeng Lai, Zhuoheng Li, Luca Cerny Oliveira, Joohi Chauhan, Brittany N Dugger, and Chen-Nee Chuah. Clipath: Fine-tune clip with visual feature fusion for pathology image analysis towards minimizing data collection efforts. InProceedings of the IEEE/CVF International Confer...
2023
-
[19]
Noise is an efficient learner for zero-shot vision- language models.arXiv preprint arXiv:2502.06019, 2025
Raza Imam, Asif Hanif, Jian Zhang, Khaled Waleed Dawoud, Yova Kementched- jhieva, and Mohammad Yaqub. Noise is an efficient learner for zero-shot vision- language models.arXiv preprint arXiv:2502.06019, 2025
2025 arXiv
-
[20]
Meduna: Language guided unsupervised adaptation of vision-language models for medical image classification.arXiv preprint arXiv:2409.02729, 2024
Umaima Rahman, Raza Imam, Dwarikanath Mahapatra, and Boulbaba Ben Amor. Meduna: Language guided unsupervised adaptation of vision-language models for medical image classification.arXiv preprint arXiv:2409.02729, 2024
2024 arXiv
-
[21]
Learning disentangled representations in the imaging domain.Medical Image Analysis, 80:102516, 2022
Xiao Liu, Pedro Sanchez, Spyridon Thermos, Alison Q O’Neil, and Sotirios A Tsaftaris. Learning disentangled representations in the imaging domain.Medical Image Analysis, 80:102516, 2022
2022
-
[22]
Domain adaptation and generalization of func- tional medical data: A systematic survey of brain data.ACM Computing Surveys, 56(10):1–39, 2024
Gita Sarafraz, Armin Behnamnia, Mehran Hosseinzadeh, Ali Balapour, Amin Meghrazi, and Hamid R Rabiee. Domain adaptation and generalization of func- tional medical data: A systematic survey of brain data.ACM Computing Surveys, 56(10):1–39, 2024
2024
-
[23]
Shenzhen hospital chest x-ray (cxr) set
Shenzhen Hospital. Shenzhen hospital chest x-ray (cxr) set. https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray- Datasets/Shenzhen-Hospital-CXR-Set/index.html
-
[24]
Montgomery county chest x-ray (cxr) set
Montgomery County. Montgomery county chest x-ray (cxr) set. https://data.lhncbc.nlm.nih.gov/public/Tuberculosis-Chest-X-ray- Datasets/Montgomery-County-CXR-Set/MontgomerySet/index.html
-
[25]
Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research.Data, 3(3):25, 2018
Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Desh- mukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research.Data, 3(3):25, 2018
2018
-
[26]
Isic archive: International skin imaging collaboration dataset
International Skin Imaging Collaboration. Isic archive: International skin imaging collaboration dataset. https://www.isic-archive.com/
-
[27]
On the robustness of medical vision-language models: Are they truly generalizable?arXiv preprint arXiv:2505.15425, 2025
Raza Imam, Rufael Marew, and Mohammad Yaqub. On the robustness of medical vision-language models: Are they truly generalizable?arXiv preprint arXiv:2505.15425, 2025. Decoupling Clinical and Class-Agnostic Features under Shift 11
2025 arXiv
-
[28]
Amend to alignment: Decoupled prompt tuning for mitigating spuri- ous correlation in vision-language models
Jie Zhang, Xiaosong Ma, Song Guo, Peng Li, Wenchao Xu, Xueyang Tang, and Zicong Hong. Amend to alignment: Decoupled prompt tuning for mitigating spuri- ous correlation in vision-language models. InForty-first International Conference on Machine Learning
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.