Pith. sign in

REVIEW 4 major objections 6 minor 17 references

Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Four CLIP-based zero-shot anomaly detection methods fail to reach clinically usable segmentation quality on brain metastasis MRI, with all mean 3D Dice scores below 50% even after adaptation to the BraTS-MET training set.

desk verdict Useful and honest CLIP-ZSAD benchmark on brain metastases, but the reported Dice/F1-max and sub-0.5 AUROC discrepancies need reconciliation before the negative result is fully trustworthy. read the letter →

arxiv 2411.09310 v1 pith:GJVYOSQB submitted 2024-11-14 cs.CV

Aldo Marzullo , Marta Bianca Maria Ranzini This is my paper · ORCID
classification cs.CV
keywords anomalydetectionzero-shotlearningCLIPmedicalimagingbrainmetastasissegmentationBraTS-METdomaingeneralizationvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether CLIP-based zero-shot anomaly detection (ZSAD), which performs well on industrial defect inspection, is ready for clinical brain imaging. The authors benchmark four CLIP-based methods—AnomalyCLIP, VAND, AnoVL, and AdaCLIP—on brain metastasis segmentation using the BraTS-MET dataset, under four setups ranging from industrial pretraining to training adapters on the medical data with a biomedical CLIP backbone. Every configuration yields mean 3D Dice scores below 50%, with the best result at 0.35, and the authors argue that high AUROC values in prior work overstate localization quality because metastases occupy very few voxels. The conclusion is that CLIP-based ZSAD is not yet clinically usable for brain metastasis detection and requires substantial domain-specific adaptation.

What carries the argument

The shared mechanism is the patch-level cosine-similarity anomaly map: for each image patch, the model compares the patch embedding's similarity to a text prompt describing an anomalous state against a prompt describing a normal state, producing a per-pixel score $M_{i,j}$. The four evaluated methods are variants of this idea: AnomalyCLIP learns object-agnostic text prompts, VAND adds trainable linear layers and memory banks of reference images, AnoVL introduces value-wise attention and test-time adaptation, and AdaCLIP combines static and per-image dynamic prompts with a semantic fusion module. The benchmark stacks the per-slice maps into a 3D volume and evaluates them with Dice, sensitivity, specificity, PPV, AUROC, and F1-max, keeping default hyperparameters and only changing the input size to $240 \times 240$.

What would settle it

If an independent evaluation with specialized metastasis prompts, a 3D-aware aggregation, or a different CLIP-like backbone produced a mean 3D Dice above 0.50 on the same BraTS-MET test split without training on BraTS labels, the claim that CLIP-based ZSAD is not ready would be falsified for that configuration; conversely, reproducing the below-0.50 result under those improved conditions would confirm the paper's conclusion.

Watch

Extended reading notes

Core claim

The central claim is a negative result: on axial T2-weighted slices from BraTS-MET, none of the four CLIP-based zero-shot anomaly detection methods reaches clinically acceptable segmentation quality. The best mean 3D Dice is 0.35 (AdaCLIP trained from scratch on BraTS), and switching the backbone to PMC-CLIP, a biomedical-pretrained vision-language model, does not fix the problem, with VAND reaching 0.32 and AnomalyCLIP 0.16. The paper further claims that AUROC is an inflated performance estimate for this setting because the anomaly class is extremely sparse, so pixel-level Dice or F1 should be the primary metric. In the authors' reading, the gap between industrial and medical performance shows that CLIP embeddings and the current prompt and architecture designs do not yet capture the variability of brain metastases.

Load-bearing premise

The negative conclusion assumes that the particular evaluation choices—default industrial prompts, unchanged hyperparameters, axial T2 slices, and 2D-to-3D slice stacking—are representative of what CLIP-based zero-shot anomaly detection can achieve on this medical task.

Editorial extensions

If this is right

  • CLIP-based zero-shot anomaly detection should not be deployed for brain metastasis screening or segmentation until domain-specific adaptation closes the gap.
  • Benchmark reports for medical ZSAD should report pixel-level Dice or F1-max alongside AUROC, since AUROC alone can hide poor localization on sparse lesions.
  • Medical-domain pretraining, such as PMC-CLIP, is not a sufficient substitute for task-specific adaptation on complex 3D data.
  • The 2D-to-3D stacking strategy is a likely source of error; volume-aware models are a concrete design direction the paper identifies.
  • Prompt design is an under-explored variable; the paper notes its default prompts may have disadvantaged the methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's implicit diagnosis would be to optimize prompt templates and hyperparameters for brain metastasis MRI while keeping the same adapters; if Dice then crosses 0.5, the bottleneck is the transfer setup rather than CLIP's embeddings.
  • The near-universal pattern of low Dice with high sensitivity suggests these models may be detecting image-level distribution shift rather than localized lesions, which would make them unsuitable for screening even when image-level AUROC looks high.
  • The peripheral-brain drop in Dice points to a spatial bias that could be probed by reweighting slices or using attention at the volume level, an extension the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates four CLIP-based zero-shot anomaly detection methods (AnomalyCLIP, VAND, AnoVL, AdaCLIP) on brain metastasis segmentation using the BraTS-METS 2023 dataset, under four training setups (industrial-pretrained, BraTS-finetuned, BraTS-from-scratch, and PMC-CLIP-backbone). The authors report Dice scores below 0.5 for all methods and conclude that CLIP-based ZSAD models are not yet suitable for clinical use on this 3D MRI task. The evaluation uses only an internal 70/30 split of the BraTS training set, with no independent test cohort, and a subsampled training set for AdaCLIP. The paper includes per-subject slice-level analyses, spatial error distributions, and correlation analyses. I find the negative claim directionally plausible but not fully supported by the paper's own metrics, because the reported F1-max values exceed 0.5 for several configurations and multiple AUROC values fall below 0.5, suggesting score-direction problems that are not discussed.

Significance. If the central negative claim were rigorously established, the paper would provide a useful falsification data point for the medical ZSAD community: it benchmarks four published methods on a clinically relevant 3D MRI dataset using open-source implementations and reports multiple segmentation metrics. This is valuable because most ZSAD evaluations in medical imaging are limited to 2D slices and do not include a 3D volumetric assessment. However, the current presentation undermines the strength of the conclusion: the inconsistency between the reported Dice scores and the F1-max column, together with AUROC values below chance, means the paper does not yet convincingly demonstrate that 'CLIP-based models are not ready' across reasonable operating points. The significance is therefore contingent on reanalysis and clarification of the threshold and score-orientation issues.

major comments (4)
  1. [Section 3.4 and Table 1] The central claim that "Dice scores below 50%" for all models is not internally consistent with Table 1. Section 2.3 defines Dice as equivalent to F1, yet the F1-max column contains values above 0.5: AdaCLIP Brats has F1-max = 0.58 and VAND PMC has F1-max = 0.74. This means that at the operating point that maximizes F1/Dice, these models exceed the 50% threshold. The reported Dice column appears to be computed at a single, unspecified threshold; the paper gives no clinical or methodological rationale for that threshold. The blanket conclusion in Section 3.4 therefore overstates the evidence. Please specify the threshold used for the Dice column, report Dice at the F1-max operating point, show PR curves, or otherwise justify the chosen operating point, and revise the central claim accordingly.
  2. [Table 1] Several AUROC values are below 0.5 (VAND PMC 0.29, AdaCLIP Brats 0.35, AnomalyCLIP Finetune 0.42, AnomalyCLIP Brats 0.43, VAND Brats 0.44). An AUROC below 0.5 indicates that the anomaly scores are anti-correlated with the ground truth, which is a symptom of score inversion, label mismatch, or an implementation error rather than a benign performance level. Section 3.1 attributes only the discrepancy between AUROC and Dice to class imbalance, and does not address why several AUROC values fall below chance. This is a load-bearing issue for the benchmark's validity because it suggests that the models' anomaly maps may be systematically reversed in some configurations. Please investigate and report whether the scores were inadvertently inverted, whether the ground-truth mask convention was reversed, or whether genuine anti-correlation occurs, and re-evaluate the affected setups with corrected or explicitly oriented scores.
  3. [Section 2.1] The evaluation is performed on a single internal 70/30 split of the BraTS-METS training set, with no independent test set, no cross-validation, and no statistical significance tests. The test set contains about 50 patients, and the reported Dice scores have large standard deviations (e.g., 0.35 ± 0.30). Without confidence intervals for the means or paired tests across subjects, the claim that the performance is "consistently low" across models is not statistically substantiated. The negative conclusion would be considerably stronger if the authors reported per-subject confidence intervals, performed tests comparing methods or setups, and ideally validated on the official BraTS-METS validation or test data.
  4. [Section 2.3] AdaCLIP was trained on only 50 randomly sampled patients, while the other methods appear to have been trained on the full 70% training split (about 115 patients). This introduces a training-set size confound when comparing AdaCLIP's Brats and PMC results with the other methods, and the paper's conclusion treats all methods equally. Section 3.4 acknowledges the subsampling as a limitation, but the conclusion "consistently low performance for all models" is not justified for AdaCLIP, whose lower Dice may be partly due to the reduced training data. The authors should either train AdaCLIP on the same number of patients as the other methods, or clearly bracket any cross-method comparisons involving AdaCLIP.
minor comments (6)
  1. [Abstract] The abstract says the methods operate "with no or minimal supervision," but two of the four setups (Finetune and Brats) train the adapters on labeled BraTS data. Please clarify the supervision degree in the abstract or add a qualifying phrase.
  2. [Section 2.1] Typo: "we used the the BraTS 2023 Brain Metastases" should read "we used the BraTS 2023 Brain Metastases dataset."
  3. [Section 3.2] The text uses corrupted formatting "V ANDexhibits" and "V ANDand"; these should be "VAND exhibits" and "VAND."
  4. [Figure 2] The caption calls the plots "Boxplots," but the text describes bars for 2D Dice scores and red crosses for 3D Dice scores; the figure caption is inconsistent with the content.
  5. [Section 3.4] The phrase "Dice scores below 50%" should specify whether this refers to the mean, median, or per-model scores and at which threshold. Without this detail, the sentence is ambiguous given the F1-max values.
  6. [Appendix Table 2] The table formatting is hard to parse: values for Prompting depth, Prompting type, and K-clusters appear to be missing or misaligned for some models. Please align the columns or use a table with explicit per-model entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical evaluation against external ground truth.

full rationale

The paper is an empirical evaluation, not a derivation. It imports four CLIP-based anomaly detection methods and two backbones from prior work, runs them on the BraTS-MET dataset using the authors' open-source implementations, and compares the resulting segmentations against manual neuroradiologist ground-truth masks. The central claim that Dice scores are below 50% and not yet sufficient for clinical use is a measured outcome, not an identity. No parameter is fitted and then renamed as a prediction: the setups that train adapters on BraTS are explicitly labeled as 'Finetune', 'Brats', and 'PMC', not presented as zero-shot predictions. The only self-citation ([2], Segato, Marzullo, Calimeri, De Momi) supports the background assertion that medical training data are difficult to obtain; it is not load-bearing for the experimental conclusion. The acknowledged limitations in Section 3.4, namely slow AdaCLIP training with a downsampled cohort, suboptimal initial prompts, and the 2D-to-3D stacking strategy, weaken generalizability but do not make the benchmark circular. A separate internal-consistency concern exists: the blanket statement that Dice scores are below 50% is not reconciled with the F1-max column in Table 1, where Dice is defined as equivalent to F1 and several F1-max values exceed 0.5, such as VAND PMC at 0.74. That is a metric-interpretation issue, not a circularity issue, because the evaluation still references external labels. The paper is self-contained as a benchmark study and does not reduce to any fitted constant or self-authored citation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical evaluation, so the central claim rests on benchmark design choices rather than derived constants. The main free choices are prompt templates, the train/test split, and AdaCLIP's reduced training subset. No new entities were introduced.

free parameters (3)
  • Zero-shot prompt templates = not reported (defaults from each method's open-source code)
    The text prompts used for 'normal' and 'anomalous' states are hand-chosen and not specified in the paper; the authors admit they may be suboptimal, yet the measured performance depends directly on these prompts.
  • Train/test split ratio = 70/30 of BraTS-MET training set (approximately 115/50 patients)
    The split of the public training set into internal train and test sets is arbitrary, and no independent test set or cross-validation is used; this choice affects the reliability of the reported generalization.
  • AdaCLIP training subset size = 50 patients (random sample) for Brats and PMC setups
    To reduce computation time, AdaCLIP was trained on fewer patients than the other methods (which used the full 115-patient training split), potentially biasing its Dice scores and comparability.
assumptions (4)
  • domain assumption BraTS-MET ground truth masks for the whole tumor (WC) are accurate and appropriate as anomaly labels for evaluating detection.
    The paper uses these masks as the target without independent verification; all segmentation metric values depend entirely on label quality.
  • domain assumption Representing each 3D volume as independent 2D axial slices and stacking 2D anomaly maps yields a valid volumetric anomaly detection evaluation for these 2D methods.
    This reconstruction ignores inter-slice context and is a design choice that could systematically lower volumetric Dice scores; the authors note this as a limitation.
  • domain assumption Dice score, not AUROC, is the appropriate primary metric for assessing clinical usability in sparse lesion segmentation.
    The paper argues that AUROC is inflated by class imbalance; while reasonable, this metric preference drives the negative conclusion, and alternative metric choices could lead to different interpretations.
  • domain assumption The open-source repositories used for each method correctly implement the published AnomalyCLIP, VAND, AnoVL, and AdaCLIP algorithms.
    The evaluation depends on these implementations being faithful to the original papers, and the authors do not audit the code beyond using it with default settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?." pith.science (2026). https://pith.science/paper/GJVYOSQB

@misc{pith2026241109310,
  author       = {Pith},
  title        = {Pith review of: Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJVYOSQB}},
  note         = {Machine review of arXiv:2411.09310}
}
read the original abstract

Zero-shot anomaly detection (ZSAD) offers potential for identifying anomalies in medical imaging without task-specific training. In this paper, we evaluate CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset. Our analysis examines their ability to detect medical-specific anomalies with no or minimal supervision, addressing the challenges posed by limited data annotation. While these models show promise in transferring general knowledge to medical tasks, their performance falls short of the precision required for clinical use. Our findings highlight the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.

Figures

Figures reproduced from arXiv: 2411.09310 by the authors.

Figure 1
Figure 1. Distribution of 3D Dice scores across subjects grouped by AD method and for each training setup: industrial , finetune , brats , and pmc . For further comparison, we analyze additional metrics as reported in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Boxplots of 2D Dice Scores for each subject across all models and datasets. Red crosses represent corresponding 3D Dice Scores for each subject. BraTS Subject IDs are listed at the bottom in the same order as they appear in the corresponding bar charts, from left to right [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Mean 2D Dice Scores plotted against normalized distance from the center of the brain (0). A value of -1 corresponds to the inferior part of the brain, while 1 represents the superior part. Yellow bars indicate the number of slices with lesions at each distance. In summary, there is a trend in achieving reasonable 2D segmentation performance near the center of the brain but struggle towards the periphery. Their 3D pe… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Sample 2D slices from 3D MRI images and their corresponding whole tumor segmentation (yellow overlap) for inferior part of the brain (a, distance < 0) and superior part of the brain (b, distance > 0) [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Overlay of ground truth and model predictions for MRI segmentation, highlighting true positives (green) , false positives (red) , and false negatives (blue) . Grayscale MRI slices provide anatomical context, showcasing model accuracy and errors. All the sample results …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

  1. [1]

    M. E. Tschuchnig, M. Gadermayr, Anomaly detection in medical imaging-a mini review, in: Data Science–Analytics and Applications: Proceedings of the 4th International Data Science Conference–iDSC2021, Springer, 2022, pp. 33–38

    work page 2022

  2. [2]

    Segato, A

    A. Segato, A. Marzullo, F. Calimeri, E. De Momi, Artificial intelligence for brain diseases: A systematic review, APL bioengineering 4 (2020)

    work page 2020

  3. [3]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763

    2021

  4. [4]

    Z. Zhao, Y. Liu, H. Wu, Y. Li, S. Wang, L. Teng, D. Liu, X. Li, Z. Cui, Q. Wang, et al., Clip in medical imaging: A comprehensive survey, arXiv preprint arXiv:2312.07353 (2023)

    arXiv 2023

  5. [5]

    A. W. Moawad, A. Janas, U. Baid, D. Ramakrishnan, L. Jekel, K. Krantchev, H. Moy, R. Saluja, K. Osenberg, K. Wilms, et al., The brain tumor segmentation (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri, arXiv preprint arXiv:2306.00838 (2023)

    arXiv 2023

  6. [6]

    Q. Zhou, G. Pang, Y. Tian, S. He, J. Chen, Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection, arXiv preprint arXiv:2310.18961 (2023)

    arXiv 2023

  7. [7]

    X. Chen, Y. Han, J. Zhang, April-gan: A zero-/few-shot anomaly classification and segmentation method for cvpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place on few-shot ad, arXiv preprint arXiv:2305.17382 (2023)

    arXiv 2023

  8. [8]

    H. Deng, Z. Zhang, J. Bao, X. Li, Anovl: Adapting vision-language models for unified zero-shot anomaly localization, arXiv preprint arXiv:2308.15939 (2023)

    arXiv 2023

Show all 17 references
  1. [9]

    Y. Cao, J. Zhang, L. Frittoli, Y. Cheng, W. Shen, G. Boracchi, Adaclip: Adapting clip with hybrid learnable prompts for zero-shot anomaly detection, arXiv preprint arXiv:2407.15795 (2024)

    2024 arXiv

  2. [10]

    W. Lin, Z. Zhao, X. Zhang, C. Wu, Y. Zhang, Y. Wang, W. Xie, Pmc-clip: Contrastive language-image pre-training using biomedical documents, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2023, pp. 525–536

    2023

  3. [11]

    Jeong, Y

    J. Jeong, Y. Zou, T. Kim, D. Zhang, A. Ravichandran, O. Dabeer, Winclip: Zero-/few-shot anomaly classification and segmentation, arXiv preprint arXiv:2303.14814 (2023)

    2023 arXiv

  4. [12]

    Davis, M

    J. Davis, M. Goadrich, The relationship between precision-recall and roc curves, in: Proceedings of the 23rd international conference on Machine learning, 2006, pp. 233–240

    2006

  5. [13]

    P. B. Kanade, P. Gumaste, Brain tumor detection using mri images, Brain 3 (2015) 146–150

    2015

  6. [14]

    Zhang, Y

    S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al., Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs, arXiv preprint arXiv:2303.00915 (2023)

    2023 arXiv

  7. [15]

    L. Hua, Y. Luo, Q. Qi, J. Long, Medicalclip: Anomaly-detection domain generalization with asymmetric constraints, Biomolecules 14 (2024) 590

    2024

  8. [16]

    Zhang, M

    X. Zhang, M. Xu, D. Qiu, R. Yan, N. Lang, X. Zhou, Mediclip: Adapting clip for few-shot medical image anomaly detection, arXiv preprint arXiv:2405.11315 (2024)

    2024 arXiv

  9. [17]

    Huang, A

    C. Huang, A. Jiang, J. Feng, Y. Zhang, X. Wang, Y. Wang, Adapting visual-language models for generalizable anomaly detection in medical images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 11375–11385. A. experiment settings ...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.