REVIEW 4 major objections 5 minor 17 references
A Vision-Language Model for Focal Liver Lesion Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A text-aligned vision-language model beats standard CLIP and MedCLIP on focal liver lesion CT classification.
desk verdict A modest, honest CLIP-style application to liver lesion classification whose internal comparisons are useful, but whose headline outperformance claim over CLIP/MedCLIP rests on an unstated and likely zero-shot baseline protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is cross-entropy alignment between image and text embeddings. Class names are expanded to full labels and placed in the prompt template 'a CT scan of tumors {label}', the frozen BERT text encoder turns each prompt into a class embedding, the image encoder output is projected by a fully connected layer to the same dimension, and the softmax of cosine similarities between the image and all class embeddings is trained with cross-entropy. The frozen text encoder and prompt template are what carry the class prior; the only trainable text-side component is the projection layer, so the added text guidance costs no extra computation at inference time.
What would settle it
Fine-tune CLIP and MedCLIP on the same 3-fold MPCT-FLLs splits, with an ImageNet-initialized ResNet backbone, the same optimizer, the same 200 epochs, and the same 'a CT scan of tumors {label}' prompt template. If either fine-tuned baseline reaches or exceeds $79.27\%$ average accuracy or $0.91$ AUC, the paper's central claim that Liver-VLM outperforms them is falsified.
Extended reading notes
Core claim
Liver-VLM is a CLIP-inspired framework in which a trainable image encoder (ResNet50 or ResNet18) embeds each three-phase CT slice and a frozen BERT text encoder embeds the class prompt 'a CT scan of tumors {label}'; a trainable projection layer brings the two modalities into a shared space, and cross-entropy loss over pairwise cosine similarities aligns each image with its true class text. The paper's central claim is that this class-level text alignment yields more discriminative features for focal liver lesions than standard CLIP or MedCLIP under annotation-scarce conditions. On the MPCT-FLLs dataset, CLIP and MedCLIP collapse to two classes and hover near chance, while every Liver-VLM variant classifies all four lesion types; the best configuration reaches $79.27\pm3.06\%$ average accuracy and $0.91\pm0.03$ AUC. The authors also claim that a lighter ResNet18 backbone performs better than ResNet50 in this data-constrained setting, and they state the resulting accuracy is still insufficient for reliable clinical application.
Load-bearing premise
The load-bearing premise is that the CLIP and MedCLIP baselines were evaluated under the same training protocol as Liver-VLM; the paper does not state whether those baselines were fine-tuned or used off the shelf, so the comparison may not be apples to apples.
Editorial extensions
If this is right
- Class-level text conditioning can be applied to other small-sample medical imaging tasks with few labels.
- A lightweight ResNet18 backbone should be preferred over ResNet50 when only a few hundred training slices are available.
- Training from scratch with text alignment already outperforms the CLIP and MedCLIP baselines, so the text signal itself, not only ImageNet pretraining, drives the improvement.
- The best model's $0.91$ AUC suggests acceptable ranking for triage, while its $79.27\%$ accuracy is below what the paper considers clinically reliable.
Reading between the lines
- Editorial inference: the reported comparison may be unfair if CLIP and MedCLIP were not fine-tuned; a same-protocol fine-tuned baseline is the decisive missing experiment.
- Editorial inference: changing the prompt template could shift results, so the method's sensitivity to wording should be tested directly.
- Editorial inference: because the 489 slices come from a single institution, external multi-center data are needed to test whether text-aligned embeddings generalize.
- Editorial inference: adding a contrastive term over image pairs to the cross-entropy alignment might further separate classes whose text descriptions differ only slightly.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Liver-VLM, a CLIP-inspired model for classifying focal liver lesions (FLLs) in multi-phase CT images. The model uses a ResNet image encoder and a frozen BERT text encoder with class-level prompt embeddings, trained with a cross-entropy loss over cosine similarities. Experiments on the MPCT-FLLs dataset (489 slices, four lesion types) compare two variants (train from scratch and ImageNet fine-tuning) with ResNet50 and ResNet18 backbones against CLIP and MedCLIP baselines. The authors report that Liver-VLM outperforms CLIP and MedCLIP in accuracy and AUC, and that the lightweight ResNet18 backbone performs better than ResNet50 under limited-data conditions.
Significance. If the claims were fully supported, the work would offer a modest contribution to low-data medical image classification by showing that aligning image features with frozen text class prototypes can provide a strong inductive bias. The paper's internal comparisons (Model 1 vs. Model 2, ResNet50 vs. ResNet18) are clearly reported and internally consistent. However, the central external comparison against CLIP and MedCLIP is currently not credible because the baseline protocol is undisclosed and the reported baseline performances are characteristic of zero-shot evaluation rather than equivalent supervised training. The absence of an image-only control further prevents attribution of the gains to the text modality. The paper therefore does not yet substantiate its main claim.
major comments (4)
- [§4.3, Table 3] The protocols for the CLIP and MedCLIP baselines are not specified. The authors state that a CLIP model with ResNet50 was used as a baseline and that MedCLIP was evaluated on the same dataset, but they do not state whether these models were fine-tuned on the MPCT-FLLs training folds, used in a zero-shot fashion, or adapted with any other procedure. The reported results—CLIP at 32.80% accuracy with 0% on FNH and HEM, and MedCLIP at 30.73% with 0% on FNH and HCC—are exactly what one would expect from frozen, off-the-shelf models on an out-of-domain CT dataset. Because the Liver-VLM models are trained for 200 epochs with labels, the comparison conflates the training procedure with the model architecture. The abstract claim that Liver-VLM 'outperforms both the standard CLIP and MedCLIP models' is therefore not supported unless the baselines received equivalent supervised fine-tuning under the same splits and hyperparameters.
- [§3, §4.3 (Tables 3 and 4)] No image-only baseline is included. The proposed method aligns image embeddings with text embeddings and optimizes with cross-entropy, but the paper never trains a standard ResNet18 or ResNet50 classifier with the same training procedure, data splits, and a simple linear head (or the same projection layer without text) to measure the contribution of the text branch. Without this control, the reported improvements over CLIP/MedCLIP may be entirely due to the ResNet backbone and the supervised training scheme, and the paper's central claim that the vision-language alignment improves small-sample classification is not established.
- [§4.1, Table 1] The data split protocol is ambiguous. The text says 'In each fold, one group was designated as the test set, while the other two were used for training,' but Table 1 lists a third 'Val' subset. The authors should clarify whether the validation group is used for model selection/early stopping, and if so, how it is incorporated into the 3-fold cross-validation. This matters for the reproducibility and fairness of all reported comparisons.
- [§4.3, Tables 3 and 4] The claim that ResNet18 'enhances classification performance' over ResNet50 is based on differences between averages from only three folds, with standard deviations that overlap (e.g., ResNet50 Model 2: 74.08±4.33; ResNet18 Model 2: 79.27±3.06). The authors should report per-fold results and perform a paired statistical test (e.g., Wilcoxon signed-rank test) or otherwise demonstrate that the observed differences are not due to fold variance.
minor comments (5)
- [Abstract and §1] There is a typo in the abstract: 'pro-pose' should be 'propose', and 'classifiers' in 'which classifiers image' should be 'classifies'. The grammar in several places ('However, its reliance...', 'a detailed description of the proposed approach .') needs editing.
- [§3.4] The prompt template 'a CT scan of tumors {label}' is described, but the exact prompt strings used for each class are not listed. Since prompt wording can materially affect zero-shot or text-encoder behavior, the authors should include the final prompts for Cyst, FNH, HCC, and HEM.
- [§4.2, Table 2] Table 2 is titled 'Computation Environment' but lists no hyperparameters such as learning rate schedule, weight decay, or the number of warm-up steps. The training setup described in the text (200 epochs, batch size 32, learning rate 0.01, AdamW) is helpful, but additional details would improve reproducibility.
- [§4.3] The standard deviations for per-class accuracies are large (e.g., HEM accuracy 63.15±25.12 for ResNet50 Model 2), yet the text discusses these differences as if they were meaningful. The authors should acknowledge the high variance and avoid over-interpreting per-class changes without statistical support.
- [References] Reference [13] for MedCLIP is cited as a MICCAI 2022 paper, but the MedCLIP model has multiple versions (e.g., with ViT or CNN encoders). The authors should specify which MedCLIP checkpoint and variant was used and how it was obtained.
Circularity Check
No circularity: Liver-VLM is supervised classification with fixed class-prototype text embeddings; its claimed gains are empirical and not self-derived.
full rationale
The paper's central claim is that Liver-VLM, trained with cross-entropy loss to align image embeddings with frozen BERT text embeddings, achieves higher accuracy and AUC than CLIP and MedCLIP on the MPCT-FLLs dataset. Nothing in the method defines the predicted label in terms of the evaluation metric, nor fits a parameter and then renames it as a prediction. The text prompts are fixed class descriptions ('a CT scan of tumors {label}'), and the image encoder is trained on labeled ROI slices; this is ordinary supervised learning with fixed class prototypes. The paper's own equation (Eq. 1) is the standard cross-entropy loss, not an identity that assumes the reported result. Self-citations [4-6] appear only as background about prior pretraining work and are not used to justify the proposed model's outcome; no uniqueness theorem or ansatz is imported from the authors' prior work. The most serious concern is experimental rather than circular: Section 4.3 reports CLIP and MedCLIP baselines without stating whether they were fine-tuned under the same protocol, and their near-chance, degenerate per-class results (e.g., FNH and HEM at 0% for CLIP) suggest zero-shot evaluation. That is a validity and fair-comparison issue, not a circularity, because even if the baselines were zero-shot, Liver-VLM's result is not derived from the baselines by construction; it is an independent empirical measurement. The paper also explicitly acknowledges that accuracy remains insufficient for clinical application, which further indicates the claims are empirical rather than tautological. No load-bearing step reduces to its own input, so the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- Image encoder and projection weights =
ResNet50 or ResNet18 plus FC_T, trained on MPCT-FLLs training folds
- Learning rate =
0.01
- Batch size =
32
- Training epochs =
200
- Input image resize =
128x128
- Prompt template =
"a CT scan of tumors {label}"
- Trainable projection layer =
unspecified output dimension
assumptions (5)
- domain assumption Radiologist annotations on the MPCT-FLLs dataset are correct and consistent.
- domain assumption Representing the three CT phases as a 3-channel RGB-like input preserves diagnostically relevant information.
- domain assumption Frozen BERT embeddings of the prompt provide meaningful class targets for alignment.
- domain assumption The cross-validation split separates patients or cases so that slices from the same volume do not appear in both training and test folds.
- domain assumption Pretrained ImageNet weights are a beneficial initialization for CT images.
Cite this review
Pith. "Pith review of A Vision-Language Model for Focal Liver Lesion Classification." pith.science (2026). https://pith.science/paper/XBII4U4Y
@misc{pith2026250503350,
author = {Pith},
title = {Pith review of: A Vision-Language Model for Focal Liver Lesion Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/XBII4U4Y}},
note = {Machine review of arXiv:2505.03350}
}
read the original abstract
Accurate classification of focal liver lesions is crucial for diagnosis and treatment in hepatology. However, traditional supervised deep learning models depend on large-scale annotated datasets, which are often limited in medical imaging. Recently, Vision-Language models (VLMs) such as Contrastive Language-Image Pre-training model (CLIP) has been applied to image classifications. Compared to the conventional convolutional neural network (CNN), which classifiers image based on visual information only, VLM leverages multimodal learning with text and images, allowing it to learn effectively even with a limited amount of labeled data. Inspired by CLIP, we pro-pose a Liver-VLM, a model specifically designed for focal liver lesions (FLLs) classification. First, Liver-VLM incorporates class information into the text encoder without introducing additional inference overhead. Second, by calculating the pairwise cosine similarities between image and text embeddings and optimizing the model with a cross-entropy loss, Liver-VLM ef-fectively aligns image features with class-level text features. Experimental results on MPCT-FLLs dataset demonstrate that the Liver-VLM model out-performs both the standard CLIP and MedCLIP models in terms of accuracy and Area Under the Curve (AUC). Further analysis shows that using a lightweight ResNet18 backbone enhances classification performance, particularly under data-constrained conditions.
Reference graph
Works this paper leans on
-
[1]
Radiology, Vol.286, pp.887– 896 (2017)
Yasaka, K., et al.: Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast -enhanced CT: A preliminary study. Radiology, Vol.286, pp.887– 896 (2017)
work page 2017
-
[2]
In: Frangi A., Schnabel J., Davat- zikos C., Alberola-Ló pez C., Fichtinger G
Liang, D., et al.: Combining Convolutional and Recurrent Neural Networks for Classifica- tion of Focal Liver Lesions in Multi -Phase CT Imaging. In: Frangi A., Schnabel J., Davat- zikos C., Alberola-Ló pez C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, LNCS, vol. 7951, pp.666-675. Springer (2018)
work page 2018
-
[3]
Wang, W., et al.: Classification of Focal Liver Lesions Using Deep Learning with Fine - tuning, In: Proceedings of Digital Medicine and Image Processing (DMIP2018), pp.56 -60, (2018)
work page 2018
-
[4]
Dong, H., et. al.: Case Discrimination: Self-supervised Feature Learning for the classifica- tion of Focal Liver Lesions,” in Chen, Y. -W. et al. (eds.), Innovation in Medicine and Healthcare, Smart Innovation, Systems and Technologies (Proc. of InMed2021), pp 241 - 249, Springer (2021)
work page 2021
-
[5]
Song, J., Dong, H., Chen, Y., Lin, L., Hu, H., and Chen, Y. -W.: Deep Neural Network - Based Classification of Focal Liver Lesions Using Phase-Shuffle Prediction Pre-training. In: Chen, Y.-W., et al. (eds.) Innovation in Medicine and Healthcare. KES InMed 2023. Smart Innovation, Systems and Technologies, vol. 357, pp. 235 –243. Springer (2023) doi:10.1007...
-
[6]
Song, J., Dong, H., Chen, Y., Zhang, X., Zhan, G., Jain, R.K., Chen, Y. -W.: Early Recur- rence Prediction of Hepatocellular Carcinoma Usin g Deep Learning Frameworks with Multi-Task Pre-Training. Information 15(8), 493 (2024). doi:10.3390/info15080493
-
[7]
Desai, K., Johnson, J.: VirTex: Learning Visual Representations from Textual Annotations. In: CVPR 2021, pp. 11162–11172. IEEE, Nashville (2021)
work page 2021
-
[8]
Sariyildiz, B., Nazarieh, S., Ricci, E.: ICMLM: Image Conditioned Masked Language Mod- eling. In: ACM MM 2020, pp. 2590–2598. ACM, Seattle (2020)
work page 2020
Show all 17 references
-
[9]
In: arXiv preprint arXiv:2010.00747 (2020)
Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive Learning of Medical Visual Representations from Paired Images and Text. In: arXiv preprint arXiv:2010.00747 (2020)
2020 arXiv
-
[10]
In: ICML 2021, pp
Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. In: ICML 2021, pp. 8748–8763. PMLR, Virtual (2021)
2021
-
[11]
R., Gotmare, A., Joty, S., Xiong, C., Hoi, S
Li, J., Selvaraju, R. R., Gotmare, A., Joty, S., Xiong, C., Hoi, S. C.: BLIP: Bootstrapped Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv preprint arXiv:2201.12086 (2022)
2022 arXiv
-
[12]
arXiv preprint arXiv:2304.08485 (2023)
Liu, H., Peng, H., Yu, Z., Ma, X., Wang, J., Ma, H., Wang, Y., Wu, J., Xie, S.: Visual Instruction Tuning. arXiv preprint arXiv:2304.08485 (2023)
2023 arXiv
-
[13]
In: MICCAI 2022, LNCS, vol
Wang, Z., Yang, J., Wang, D., Xu, Y., Bai, J., Zhou, S.K.: MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. In: MICCAI 2022, LNCS, vol. 13433, pp. 40–50. Springer, Singapore (2022)
2022
-
[14]
In: CVPR 2016, pp
He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: CVPR 2016, pp. 770–778. IEEE, Las Vegas (2016)
2016
-
[15]
In: NAACL -HLT 2019, pp
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL -HLT 2019, pp. 4171–4186. ACL, Minneapolis (2019)
2019
-
[16]
In: Medical Physics, vol
Xu, Y., et al.: PA-ResSeg: A Phase Attention Residual Network for Liver Tumor Segmen- tation from Multi -phase CT Images. In: Medical Physics, vol. 48, no. 7, pp. 3752 –3766. AAPM, (2021)
2021
-
[17]
In: ICLR 2019
Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. In: ICLR 2019. Open- Review, New Orleans (2019)
2019
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.