REVIEW 2 major objections 6 minor 25 references
Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detection
T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A multiscale vision-language model improves medical out-of-distribution detection by fusing global and local patch embeddings and training on hard boundary-region pseudo-OOD patches.
desk verdict A competent applied OOD paper with a real new combination and strong reported gains, but the mechanism behind the pseudo-OOD selection is not actually validated and the ablation table is sloppy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a four-part training pipeline built on a frozen CLIP encoder. First, multi-scale inputs are created by upsampling the image and partitioning it into $n\times n$ and $2n\times 2n$ patches; the original image yields a global embedding $u^0$, while mid-scale and high-scale patches yield embeddings $\{u^1_i\}$ and $\{u^2_j\}$. Second, cross-scale visual fusion augments each higher-scale patch embedding with its containing lower-scale embedding weighted by cosine similarity (Eq. 1), so local patches inherit global context. Third, scale-wise text-side biases $b^0$ and $b^2$ (with the mid-scale bias their average) are added to the frozen text embeddings to produce per-scale class probabilities from softmax over cosine similarities (Eq. 2), and disease-irrelevant patches are filtered by an average-entropy threshold. Fourth, hard pseudo-OOD generation selects the top-K high-scale patches with highest entropy gain $\Delta H(p^2_j) = H(p^2_j) - H(p^1_{j^*})$, fuses them with lower-scale embeddings (Eq. 4), and trains the model with the outlier-exposure loss (Eq. 5) that maximizes entropy on these pseudo-OOD examples. During inference, the three scale-wise prediction distributions are averaged and the maximum softmax probability is used as the OOD score.
What would settle it
A direct test would be to take a held-out medical dataset, run the pretrained CLIP encoder on normal tissue, lesion interior, and lesion-boundary patches, and inspect their softmax entropy distributions: if normal-tissue patches are not consistently the highest-entropy, or if boundary patches do not show a larger entropy jump than their containing mid-scale patch, then the selected hard pseudo-OOD patches in Eq. (4) are not actually boundary examples, and an ablation replacing the entropy-gain selector with random patch selection should produce comparable results.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the combination of cross-scale visual fusion and cross-scale hard pseudo-OOD generation is what makes a CLIP-based OOD detector competitive for medical near-OOD data. By upsampling and partitioning each image into global, mid-scale and high-scale patches and coupling their embeddings with cosine-weighted context, the model keeps fine detail while seeing the whole image; by selecting high-scale patches with the largest entropy gain relative to their containing mid-scale patch and fusing them with lower-scale information, the model trains against pseudo-OOD examples that sit close to real lesions. In the reported experiments, HVL reaches an average FPR95 of 56.05% and AUROC of 83.34% across the three benchmarks, and the ablations show that removing either the fusion or the entropy-gain pseudo-OOD mechanism degrades detection, while replacing the entropy-gain selector with background-region selection also hurts.
Load-bearing premise
The pseudo-OOD signal rests on the assumption that, under the pretrained CLIP encoder, normal-tissue patches are more uniformly uncertain (higher softmax entropy) than disease patches, and that the small patch straddling a lesion's boundary has higher entropy than the larger mid-scale patch containing it; if normal tissue is confidently classified or lesion borders do not produce that entropy gap, the entropy-gain selector stops selecting hard boundary examples and the OOD gains are expected to shrink or vanish.
Editorial extensions
If this is right
- HVL's reported numbers on NCT-CRC, ISIC-4 and Skin40 imply that VLM-based OOD detection in medicine should be built on multi-scale visual inputs rather than single global or local embeddings.
- The entropy-gain selection rule is shown to generalise from dermatology to histology, suggesting the boundary-patch pseudo-OOD signal transfers across medical imaging domains.
- Because the method encodes original images and patches in parallel during inference, the paper reports latency nearly identical to same-backbone models and only modest memory overhead, making the approach practical for clinical deployment.
- Ablations support that cross-scale fusion and hard pseudo-OOD generation contribute independently: each alone improves detection, and their combination gives the best FPR95 and AUROC.
Reading between the lines
- We infer that the entropy-gain heuristic is essentially a boundary-aware label-free segmentation signal: it tags the spatial transition from confidently classified lesion tissue to uncertain normal tissue, so the same idea could be adapted to non-medical near-OOD settings where foreground-background transitions carry semantic information (e.g., satellite imagery, defect inspection).
- We infer that the method's sensitivity to the partition count $n$ (performance drops when $n>4$) points to an implicit optimal patch size related to typical lesion scale; a testable extension is to make $n$ or the patch grid adaptive per image, which the paper does not explore.
- We infer that because all pseudo-OOD embeddings are mined from ID images alone, the framework could in principle be applied to unlabeled ID data or to a new clinical site without OOD labels, providing a data-efficient route to OOD-aware calibration before deployment.
- We infer that the cosine-weighted fusion acts as a fixed spatial attention; replacing the hand-set cosine weights with learned attention or a graph over patches is a natural, untested variant that could either improve or simplify the method.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents HVL, a hierarchical vision-language framework for out-of-distribution (OOD) detection in medical images. It partitions each image into multi-scale patches, encodes them with a frozen CLIP image encoder, and fuses the patch embeddings in a coarse-to-fine manner by adding cosine-weighted context from lower-scale embeddings. Learnable biases are added to the text embeddings for scale-wise alignment, and patches whose softmax entropy is above the average are discarded as disease-irrelevant. For hard pseudo-OOD generation, high-scale patches with the largest entropy gain relative to their parent mid-scale patch are selected, then augmented with lower-scale embeddings, and trained with an outlier-exposure entropy-maximization loss. Classification is based on the averaged scale-wise prediction distributions, and the maximum softmax probability (MSP) is used for OOD detection at inference. Experiments on Skin40, NCT-CRC, and ISIC-4 report lower FPR95 and higher AUROC than all compared methods, with ablations and sensitivity analyses. The source code is made publicly available.
Significance. If the reported results are reproducible, HVL is a meaningful contribution: it targets the practically important setting of near-OOD detection in medical imaging; it provides a public implementation; it reports standard deviations and multiple ablations; and it consistently outperforms both uni-modal and VLM-based baselines on the three benchmarks. The cross-scale fusion and pseudo-OOD augmentation ideas are clearly described and the empirical story is coherent. The main caveats are that the mechanism attributed to the entropy-gain selection—identifying lesion-boundary patches—is not directly verified, and the concurrent GLAli method is cited but not compared, so the state-of-the-art claim should be treated with caution until those points are addressed.
major comments (2)
- [Section 2.3, Eq. (4) (with Eq. (1))] The entropy-gain criterion ΔH(p2_j) = H(p2_j) - H(p1_j*) is computed from the fused embeddings \hat u2_j and \hat u1_j* defined in Eq. (1), not from the raw patch embeddings. Because \hat u1_j* already contains the global embedding u0 added with cosine weighting, a parent patch of normal tissue can have artificially low entropy whenever global context aligns with a confidently classified class; the '-H(p1_j*)' term may then reflect the amount of injected context rather than the presence of lesion tissue. Similarly, \hat u2_j contains its parent context, so the entropy gap mixes local tissue statistics with scale-dependent context injection. The ablation in Table 2 (row 3 versus row 2) only contrasts ΔH selection with background-region selection and does not control for this confound. Please add an experiment that computes the entropy-gain selection on unfused embeddings or with the fusion ablated, or that visualizes the selected patches, in order to validate the claimed boundary-region mechanism. Without such evidence, the central rationale for the hard pseudo-OOD generation strategy is not established.
- [Section 3.1, Table 1] The paper explicitly identifies GLAli [7] as concurrent work on VLM-based few-shot OOD detection with multi-scale image-text alignment, and the method description says 'Unlike GLAli, our method...', but GLAli is not included in the comparison in Table 1. Since the abstract and Section 3.2 claim superior OOD detection over existing methods, the absence of this directly related baseline makes the state-of-the-art claim incomplete. Please add the GLAli comparison under the same protocol, or state clearly why it cannot be evaluated.
minor comments (6)
- [Section 2.2] The threshold H^1 is defined as the sum Σ_i H(p1_i) but is described as the average entropy and used as a threshold ('higher than H^1'); this is internally inconsistent and should be corrected to (1/(n×n))Σ_i H(p1_i).
- [Section 3.2] The term 'AURCO' should be 'AUROC'.
- [Table 2] The row labels and the references to 'row 2 and row 4' in the text are difficult to follow; please explicitly name the configuration of each row in the caption or table body.
- [Section 2.2] The sentence 'such disease-relevant in general should not be used to recognize the disease category' is ungrammatical; presumably 'disease-irrelevant' patches should not be used.
- [Section 3.2, Inference cost] With n=2, the method encodes the original image, 4 mid-scale patches, and 16 high-scale patches, i.e., 21 forward passes per image; the statement that latency is 'nearly identical to same-backbone models' is surprising and should be substantiated with end-to-end timing measurements, not only memory usage.
- [Figure 2] The caption notes that ISIC-4 results are omitted due to space constraints; please include them in the supplementary material for completeness.
Circularity Check
No significant circularity: evaluation is on held-out OOD data and the pseudo-OOD training signal is self-training, not a re-fitted prediction.
full rationale
HVL's claimed derivation chain is an empirical training pipeline, not a derivation that smuggles its target into its inputs. The cross-scale fusion (Eq. 1) and hard pseudo-OOD generation (Eqs. 4-5) operate on ID training images only. The pseudo-OOD embeddings q^k are selected by entropy gain computed from the model's own softmax probabilities on ID patches (Section 2.3), and LOOD maximizes entropy on those selected embeddings; this is a self-training / outlier-exposure objective, not a fitted parameter that is later renamed as a prediction. The reported OOD metrics (FPR95, AUROC in Table 1) are computed on held-out OOD test sets, and no test OOD data or test scores are used to select K, fit biases, or choose the entropy-gain threshold, so there is no fitted-input-called-prediction or self-definitional step. The only self-citations are the concurrent-work reference [7] (with overlapping authors) in the introduction, which is not load-bearing for any result. Whether the entropy-gain heuristic truly isolates lesion boundaries is an empirical/confound question, but it is a correctness risk, not circularity. The paper's central improvement is therefore self-contained against external CLIP checkpoints and public benchmarks.
Assumptions & free parameters
free parameters (3)
- n (patch partition count per dimension) =
2
- K (number of selected hard pseudo-OOD high-scale patches) =
4
- tau (softmax temperature) =
not reported
assumptions (4)
- domain assumption Frozen CLIP ViT-B/16 features provide a valid embedding space for medical images.
- domain assumption Disease-irrelevant patches have higher softmax entropy than lesion-containing patches.
- domain assumption Entropy gain H(p2_j) - H(p1_j*) identifies high-scale patches at lesion boundaries.
- domain assumption Outlier exposure with pseudo-OOD samples improves OOD discrimination.
Cite this review
Pith. "Pith review of Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detection." pith.science (2026). https://pith.science/paper/IRPY3VU4
@misc{pith2026250817667,
author = {Pith},
title = {Pith review of: Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/IRPY3VU4}},
note = {Machine review of arXiv:2508.17667}
}
read the original abstract
In trustworthy medical diagnosis systems, integrating out-of-distribution (OOD) detection aims to identify unknown diseases in samples, thereby mitigating the risk of misdiagnosis. In this study, we propose a novel OOD detection framework based on vision-language models (VLMs), which integrates hierarchical visual information to cope with challenging unknown diseases that resemble known diseases. Specifically, a cross-scale visual fusion strategy is proposed to couple visual embeddings from multiple scales. This enriches the detailed representation of medical images and thus improves the discrimination of unknown diseases. Moreover, a cross-scale hard pseudo-OOD sample generation strategy is proposed to benefit OOD detection maximally. Experimental evaluations on three public medical datasets support that the proposed framework achieves superior OOD detection performance compared to existing methods. The source code is available at https://openi.pcl.ac.cn/OpenMedIA/HVL.
Figures
Reference graph
Works this paper leans on
-
[7]
Jie, Y., Xiaoyuan, G., Wei-Shi, Z., Hao, C., Ruixuan, W.: Global and local vision- language alignment for few-shot learning and few-shot ood detection. In: MICCAI (2025)
work page 2025
-
[1]
Bai, Y., Han, Z., Cao, B., Jiang, X., Hu, Q., Zhang, C.: Id-like prompt learning for few-shot out-of-distribution detection. In: CVPR (2024)
work page 2024
-
[2]
In: ICLR (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
2021
-
[3]
Fang, K., Tao, Q., Lv, K., He, M., Huang, X., Yang, J.: Kernel PCA for out-of- distribution detection. In: NeurIPS (2024)
work page 2024
-
[4]
Fu, W., Chen, Y., Liu, W., Yue, X., Ma, C.: Evidence reconciled neural network for out-of-distribution detection in medical images. In: MICCAI (2023)
work page 2023
-
[5]
In: ICLR (2017)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of- distribution examples in neural networks. In: ICLR (2017)
2017
-
[6]
Hendrycks, D., Mazeika, M., Dietterich, T.G.: Deep anomaly detection with outlier exposure. In: ICLR (2019)
work page 2019
-
[8]
Kather, J.N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.A., Gaiser, T., Marx, A., Valous, N.A., Ferber, D., et al.: Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine (2019)
work page 2019
Show all 25 references
-
[9]
In: ICLR (2015) 10 R
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015) 10 R. Lai et al
2015
-
[10]
In: ECCV (2024)
Lafon, M., Ramzi, E., Rambour, C., Audebert, N., Thome, N.: Gallop: Learning global and local prompts for vision-language models. In: ECCV (2024)
2024
-
[11]
In: CVPR (2024)
Li, T., Pang, G., Bai, X., Miao, W., Zheng, J.: Learning transferable negative prompts for out-of-distribution detection. In: CVPR (2024)
2024
-
[12]
In: MICCAI (2020)
Li, Z., Zhong, C., Wang, R., Zheng, W.S.: Continual learning of new diseases with dual distillation and ensemble strategy. In: MICCAI (2020)
2020
-
[13]
In: CVPR (2023)
Liu, X., Lochman, Y., Zach, C.: GEN: pushing the limits of softmax-based out-of- distribution detection. In: CVPR (2023)
2023
-
[14]
In: MICCAI (2022)
Mehta, D., Gal, Y., Bowling, A., Bonnington, C.P., Ge, Z.: Out-of-distribution detection for long-tailed and fine-grained skin lesion images. In: MICCAI (2022)
2022
-
[15]
In: NeurIPS (2023)
Miyai, A., Yu, Q., Irie, G., Aizawa, K.: Locoop: Few-shot out-of-distribution de- tection via prompt learning. In: NeurIPS (2023)
2023
-
[16]
In: ICLR (2024)
Nie, J., Zhang, Y., Fang, Z., Liu, T., Han, B., Tian, X.: Out-of-distribution detec- tion with negative prompts. In: ICLR (2024)
2024
-
[17]
Oh, J., Falahkheirkhah, K., Bhargava, R.: Are we ready for out-of-distribution detection in digital pathology? In: MICCAI (2024)
2024
-
[18]
In: ICML (2021)
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)
2021
-
[19]
In: ECCV (2016)
Sun, X., Yang, J., Sun, M., Wang, K.: A benchmark for automatic visual classifi- cation of clinical skin disease images. In: ECCV (2016)
2016
-
[20]
In: ICLR (2023)
Tao, L., Du, X., Zhu, J., Li, Y.: Non-parametric outlier synthesis. In: ICLR (2023)
2023
-
[21]
Scientific data (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data (2018)
2018
-
[22]
In: CVPR (2022)
Wang, H., Li, Z., Feng, L., Zhang, W.: Vim: Out-of-distribution with virtual-logit matching. In: CVPR (2022)
2022
-
[23]
In: NeurIPS (2024)
Yu, G., Zhu, J., Yao, J., Han, B.: Self-calibrated tuning of vision-language models for out-of-distribution detection. In: NeurIPS (2024)
2024
-
[24]
arXiv preprint arXiv:2409.04796 (2024)
Zeng, F., Cheng, Z., Zhu, F., Zhang, X.Y.: Enhancing outlier knowledge for few- shot out-of-distribution detection with extensible local prompts. arXiv preprint arXiv:2409.04796 (2024)
2024 arXiv
-
[25]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition
Zhou, Y.: Rethinking reconstruction autoencoder-based out-of-distribution detec- tion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition. pp. 7379–7387 (2022)
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.