Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A CLIP-based vision-language model adapted with LoRA, multi-level CLS fusion, and spherical feature interpolation reaches 95% classification accuracy and top retrieval on ENT endoscopy images, as reported on the ENTRep public test set.

desk verdict A plausible challenge entry whose text-to-image retrieval claim is not verifiable because the queries are never defined; the classification part is solid engineering, the retrieval part needs revision. read the letter →

arxiv 2509.00752 v1 pith:XPIIMNMI submitted 2025-08-31 cs.CV

Y Hop Nguyen , Doan Anh Phan Huu , Trung Thai Tran , Nhat Nam Mai , Van Toi Giap , Thao Thi Phuong Dao , Trung-Nghia Le This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelsmedicalimageanalysisENTendoscopycontrastivelearningmultimodalretrievalCLIPLoRAlow-resourceimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that a single vision-language model, starting from CLIP's pretrained image-text alignment, can be adapted to a small ENT endoscopy dataset and simultaneously do three things: label an image, retrieve similar images, and find images from a text description. The adaptation recipe is Low-Rank Adaptation on the image encoder, aggregation of CLS tokens from several layers through a small fusion transformer, spherical interpolation between same-class features, and fixed class-specific prompts. On the public ENTRep test set the model reaches 95% accuracy and F1 for classification, Recall@1 of 0.93 for image-to-image retrieval and 0.92 for text-to-image retrieval; on the private split the numbers are 0.91 accuracy and 0.89/0.90 Recall@1. The ablation shows each piece contributes: baseline CLIP accuracy of 0.84 rises to 0.88 with LoRA, 0.93 with fusion, and 0.95 with spherical augmentation. This matters because endoscopy interpretation in low-resource clinical settings needs cross-modal queries, not just labels, and the paper indicates that can be done without heavy domain-specific text training.

What carries the argument

The central object is the multi-level CLS token fusion (MFA). In a vision transformer, the CLS token is the special output vector that summarizes the whole image; the paper takes the CLS tokens from early, middle, and late layers and feeds them to a lightweight fusion transformer, producing a 'CLSfusion' embedding that mixes fine-grained texture with high-level anatomy. Around this, LoRA (low-rank weight updates on attention projections) keeps fine-tuning cheap, spherical interpolation (Slerp) augments same-class features along the unit hypersphere, and frozen text embeddings from class-specific prompts drive the contrastive loss. The fusion is the piece that lets one embedding carry both lo

What would settle it

Have a second ENT specialist independently re-label the test images, especially the left/right and vocal-cord classes, then re-run the published evaluation. If many labels change and the reported 0.95 accuracy and 0.93 Recall@1 drop, the central claim overstates true performance. A second check is to train the same full pipeline on 565 images from a different ENT center and see whether the numbers hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that CLIP's pretrained vision-language alignment can be transferred to ENT endoscopy with only light adaptation of the image side. The text encoder is frozen; the image encoder gets LoRA on the query/key/value projections of each ViT block, then reads out CLS tokens from multiple blocks and fuses them in a small ViT, and during training generates extra same-class embeddings by spherical linear interpolation on the unit hypersphere. Fixed prompts of the form 'A photo of a [class], Image description.' supply the text embeddings, and a weighted classification plus symmetric contrastive loss trains the whole thing. The ablation is the evidence for the mechanism: each

Load-bearing premise

The method assumes CLIP's pretrained vision-language alignment transfers to ENT endoscopy imagery through simple fixed prompts, and that the 565-image labels, including ambiguous left/right and vocal-cord states, are reliable enough for supervised contrastive learning.

Editorial extensions

If this is right

  • One trained model answers three clinical queries: classification, image-to-image retrieval, and text-to-image retrieval, so a deployment could replace three separate pipelines.
  • Frozen text encoder plus LoRA means adapting to a new small medical dataset costs few trainable parameters, making the recipe usable in hospitals without large annotated collections.
  • On this benchmark the approach beats standard CNNs and a medical-domain CLIP variant, suggesting the limiting factor is not the pretraining domain but how the image features are adapted and aggregated.
  • The remaining errors concentrate in left/right symmetric classes, so cleaning or re-annotating those labels would likely be the fastest route to higher scores.
  • The fusion module reads a ViT's layers generically, so the same recipe could be attached to other CLIP-style encoders without changing the training objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the text encoder is frozen and prompts are handcrafted, the reported text-to-image retrieval tests how well CLIP's pretrained language priors cover anatomy terms, not how well the model learned new medical text; learnable prompts or LLM-generated descriptions might change the class separations.
  • Beyond the paper: the public-to-private accuracy drop (0.95 to 0.91) and the acknowledged label ambiguity suggest that an expert re-adjudication of the benchmark could shift both absolute scores and the relative ranking of methods.
  • Beyond the paper: Slerp interpolates within a class; applying the same interpolation across classes with soft labels would separate the benefit of on-manifold augmentation from the benefit of label-preserving mixing.
  • Beyond the paper: the same architecture is a natural candidate for other small-data medical image domains or video endoscopy frames, since neither the fusion module nor SFA depends on ENT-specific structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a unified vision-language framework for ENT endoscopy image analysis, built on CLIP ViT-B/16 with three additions: LoRA-based adaptation of the image encoder, multi-level CLS token aggregation (MFA), and spherical linear interpolation-based feature augmentation (SFA). The model is trained with a weighted classification plus image-text contrastive loss, using handcrafted class prompts, and is evaluated on the ACM MM'25 ENTRep challenge for image classification, image-to-image retrieval, and text-to-image retrieval. The authors report 95% accuracy/F1 on the public test set, 0.93/0.89 Recall@1 for image-to-image retrieval, and 0.92/0.90 Recall@1 for text-to-image retrieval on public/private test sets, with ablations attributing gains to each component. The central claim is that this architecture achieves state-of-the-art performance on a small medical dataset while unifying three tasks. The results are plausible, but several load-bearing details are underspecified and one reported ablation is internally inconsistent.

Significance. If the claims hold, the work would be a useful demonstration that a CLIP backbone with lightweight adaptation can be competitive on a small, domain-specific medical dataset and can support both classification and retrieval. The paper gives credit to prior work, includes a candid limitations section (Section 7) acknowledging label ambiguity and small dataset size, and provides backbone comparisons and qualitative t-SNE/Grad-CAM analyses. These are strengths. However, the central retrieval results are currently not verifiable because the text-to-image query protocol is not defined, and the ablation study contains a direct numerical contradiction that prevents causal attribution of the reported gains. The method also omits several architectural and training details needed for reproducibility. The contribution is potentially significant, but the manuscript in its current form does not adequately support the headline numbers.

major comments (4)
  1. [Section 3.7 and Eq. (1)] The text-to-image retrieval protocol is never specified. Section 3.7 states that 'we extract feature representation of the text query using the CLIP text encoder (Equation 10)', but the only text embeddings described in the paper are the seven fixed class prompts of Eq. (1), and these same prompts are used as the positives in the contrastive loss Eq. (12). If the test queries are those class prompts, then the reported Recall@1 and MRR measure nearest-class-centroid ranking over the seven classes, not cross-modal retrieval of free-text descriptions. If the test queries are instead the dataset's DescriptionEN strings, the paper does not describe how they are encoded, normalized, batch-paired, or evaluated, and Section 7 concedes that these annotations are 'inconsistent or loosely defined'. Either way, the text-to-image retrieval claim in Tables 1-3 and the abstract is not verifiable from t
  2. [Section 6.1, Table 4, and Table 5] The ablation baseline accuracy is reported inconsistently. The prose in Section 6.1 states that the baseline achieves 'Accuracy: 0.75, Recall@1 for image-to-image: 0.85, and text-to-image: 0.84', but the corresponding row of Table 4 reports baseline Accuracy 0.84 (with i2i 0.85 and t2i 0.84). Table 5 also lists CLIP-ViT-B/16, which should be the same baseline, with Accuracy 0.84. This is not a typo in isolation: it changes the reported improvement from LoRA (0.88 vs 0.75, i.e., +0.13) to 0.88 vs 0.84 (+0.04), and similarly affects the attribution to MFA and SFA. Because the central claim is the incremental benefit of each component, this discrepancy is load-bearing. Please reconcile the numbers, report the exact training/evaluation configuration of the baseline, and ideally provide confidence intervals, since the test set is small and the differences are only a few images.
  3. [Section 3.4, Eqs. (6)-(8)] The multi-level feature aggregation module is under-specified. Eq. (7) defines Fmulti as a concatenation of CLSfusion and CLS tokens from K selected layers, but the paper does not state how many layers are selected (K), which layer indices are used, or why those layers were chosen. The FusionViT module is described only as 'a lightweight Vision Transformer Model' with no number of layers, hidden dimension, input/output projection, or training details. Since one of the paper's claimed contributions is that multi-level CLS aggregation improves accuracy and retrieval (Section 6.1), the reader cannot reproduce the method or distinguish the effect of MFA from simply adding an extra transformer on top of the final CLS token. Please provide the specific configuration and, if possible, an ablation varying K and layer depth.
  4. [Section 3.5 and Eq. (9)] The integration of spherical feature augmentation into training is not described. Eq. (9) defines the interpolated feature f_aug, but the paper never states how f_aug is used: is it added to the contrastive batch, used as a positive pair, fed into the classifier, or used only as a regularizer? The interpolation coefficient lambda is listed as a hyperparameter but its value is not given in Section 4.1 (only LoRA rank, alpha, mu1, mu2 are reported). There is also no description of how pairs (f1, f2) are sampled from 'the same class' or whether interpolation is performed on training embeddings or on a separate feature bank. Without this, the SFA component is not reproducible and the reported 0.95 vs 0.93 accuracy gain from SFA cannot be evaluated.
minor comments (6)
  1. [Table 5] The ResNet-50 row reads '078' instead of '0.78'. Please correct the typo.
  2. [Section 3.6, Eqs. (11)-(13)] The contrastive loss notation is ambiguous: u_i is the text embedding of class c_j, but the paper does not state how an image x_i is paired with a text u_i inside a batch, nor whether there is exactly one text per class per batch. Please define the batch construction and the pairing rule.
  3. [Tables 1-3] The column heading 'Top' is unexplained. If these are challenge submission ranks or top-k settings, they should be defined; otherwise the column should be removed.
  4. [Figure 1 caption] The caption says 'while used for the classification task', which appears to be an incomplete sentence. Please rephrase.
  5. [Section 4.1] The text says 'We evaluate on public datatest' in Section 6.1; there is a missing space and the term 'datatest' is nonstandard. Also, the hyperparameters for the FusionViT and SFA modules are not listed in the training setup.
  6. [References] Reference [11] has a typo in the arXiv number ('arXiv2508.04801' should be 'arXiv:2508.04801'). Some other references use inconsistent formatting (e.g., 'PmLR' vs 'PMLR').

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; held-out evaluation and frozen text encoder keep the derivation self-contained.

full rationale

The paper's claimed derivation chain is: take a frozen CLIP text encoder; define class-specific prompts (Eq. 1); train a LoRA-adapted image encoder with a contrastive loss against those text embeddings plus a classification head (Eq. 14); evaluate on public/private held-out test splits of the ENTRep Challenge. This is a conventional supervised fine-tuning pipeline. The test labels and test images are external to the training set, so classification accuracy, image-to-image recall, and text-to-image recall are predictions on unseen data, not re-statements of the training objective. The only potentially circular element is text-to-image retrieval: Section 3.7 says text queries are encoded via Eq. (10), which only defines embeddings of the seven class prompts from Eq. (1); if the challenge queries were exactly those prompts, the t2i retrieval would be a class-prototype ranking and hence a relabeled classification. However, the paper does not state that queries are these prompts, and the challenge may define Track 3 queries separately; the ambiguity is a specification gap, not a demonstrated circular reduction. Section 7's admission that DescriptionEN annotations are "inconsistent or loosely defined" further weakens the cross-modal story but does not make the derivation circular. The self-citation of the dataset paper [11] (with one overlapping author) is not load-bearing: the benchmark is public and the test set is external. The ablation prose/table inconsistency (Section 6.1 says baseline accuracy 0.75, Table 4 says 0.84) is an internal reporting error, not circularity. Overall, no step in the derivation assumes the conclusion or fits a parameter to the reported test numbers.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physics or formal entities. Its load-bearing assumptions are standard transfer-learning and data-labeling assumptions, plus a handful of hand-set hyperparameters that the reported metrics depend on.

free parameters (5)
  • LoRA rank r and alpha = r=4, alpha=8 (scaling gamma=2)
    Hand-chosen hyperparameters that control the capacity of the LoRA adaptation (Section 4.1). The central results depend on these values.
  • Loss weights mu1, mu2 = mu1=1.0, mu2=0.5
    Weights balancing classification and contrastive losses (Eq. 14, Section 4.1). These are manually set and affect the reported performance.
  • Learning rate and optimizer hyperparameters = lr=5e-4, beta=(0.9,0.999), 50 epochs
    Training configuration (Section 4.1) that influences convergence and final metrics.
  • Selection of layers for MFA = not specified
    Which K transformer layers' CLS tokens are aggregated (Eq. 7) is not reported; this choice is a free parameter affecting the feature fusion and the results.
  • SFA interpolation coefficient lambda = not specified (in [0,1])
    The spherical interpolation mixing factor is not given; its distribution affects the augmented features and training dynamics.
assumptions (4)
  • domain assumption CLIP ViT-B/16 pretrained weights provide a suitable initialization for medical endoscopy images.
    The entire method is built on fine-tuning CLIP with LoRA; if CLIP's features do not transfer to endoscopy, the reported performance would not be achieved. Invoked in Section 3.1 and throughout.
  • domain assumption The ENTRep dataset's 565 images and their labels (class, type, descriptions) are accurate enough for supervised contrastive learning.
    The paper itself notes label ambiguity for left/right classes and vc-open/closed (Section 7). The central results rest on the noisy labels being sufficient for training.
  • standard math The contrastive loss and cross-entropy loss combination (Eq. 14) is a valid objective for aligning image and text representations.
    This is a standard deep learning objective; no proof is needed but it is an assumption that the combination works for this data.
  • domain assumption The public and private test splits of the challenge are representative of the underlying distribution.
    The paper reports results on these splits without describing how they were constructed; if the splits are easy or leaked, the numbers would not be meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification." pith.science (2026). https://pith.science/paper/XPIIMNMI

@misc{pith2026250900752,
  author       = {Pith},
  title        = {Pith review of: Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPIIMNMI}},
  note         = {Machine review of arXiv:2509.00752}
}
read the original abstract

We present a unified vision-language framework tailored for ENT endoscopy image analysis that simultaneously tackles three clinically-relevant tasks: image classification, image-to-image retrieval, and text-to-image retrieval. Unlike conventional CNN-based pipelines that struggle to capture cross-modal semantics, our approach leverages the CLIP ViT-B/16 backbone and enhances it through Low-Rank Adaptation, multi-level CLS token aggregation, and spherical feature interpolation. These components collectively enable efficient fine-tuning on limited medical data while improving representation diversity and semantic alignment across modalities. To bridge the gap between visual inputs and textual diagnostic context, we introduce class-specific natural language prompts that guide the image encoder through a joint training objective combining supervised classification with contrastive learning. We validated our framework through participation in the ACM MM'25 ENTRep Grand Challenge, achieving 95% accuracy and F1-score in classification, Recall@1 of 0.93 and 0.92 for image-to-image and text-to-image retrieval respectively, and MRR scores of 0.97 and 0.96. Ablation studies demonstrated the incremental benefits of each architectural component, validating the effectiveness of our design for robust multimodal medical understanding in low-resource clinical settings.

Figures

Figures reproduced from arXiv: 2509.00752 by the authors.

Figure 1
Figure 1. The overall architecture of our proposed CLIP-based multi-modal framework for ENT endoscopy analysis. The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Grad-CAM visualization highlights the important [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. t-SNE visualization of the features extracted from [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages

  1. [1]

    Federico Baldassarre, Diego González Morín, and Lucas Rodés-Guirao. 2017. Deep koalarization: Image colorization using cnns and inception-resnet-v2.arXiv preprint arXiv:1712.03400 (2017)

    arXiv 2017

  2. [2]

    Krishna Chaitanya, Naseer Karani, Christian Baumgartner, et al. 2020. Contrastive learning of global and local features for medical image segmentation with limited annotations. In NeurIPS

    work page 2020

  3. [3]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2020

  4. [4]

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org

    work page 2016

  5. [5]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778

    2016

  6. [6]

    Yu He, Xiang Peng, Jun Zhao, et al. 2021. TransFG: A Transformer Architecture for Fine-grained Recognition. In AAAI

    work page 2021

  7. [7]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  8. [8]

    Xinyu Huang, Yuhao Zhang, Zhi Chen, et al . 2021. GLoRIA: A Multimodal Global-Local Representation Learning for Medical Images and Reports. In CVPR

    work page 2021

Show all 22 references
  1. [9]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  2. [10]

    Doniyorjon Mukhtorov, Madinakhon Rakhmonova, Muksimova Shakhnoza, and Young Im Cho. 2023. Endoscopic Image Classification Based on Explainable Deep Learning. Sensors (Basel, Switzerland) 23 (2023). https://api.semanticscholar.org/ CorpusID:257620420

    2023

  3. [11]

    Nguyen, Thanh Dinh Le, and Minh-Triet Tran

    Trong-Thuan Nguyen, Viet-Tham Huynh, Thao Thi Phuong Dao, Ha Nguyen Thi, Tien To Vu Thuy, Uyen Hanh Tran, Tam V. Nguyen, Thanh Dinh Le, and Minh-Triet Tran. 2025. ACM Multimedia Grand Challenge on ENT Endoscopy Analysis. arXiv preprint arXiv2508.04801 (2025)

    2025 arXiv

  4. [12]

    Keiron O’shea and Ryan Nash. 2015. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458 (2015)

    2015 arXiv

  5. [13]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  6. [14]

    Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. 2022. TokenFusion: Facilitating Information Fusion in Token-Based Transformers for Multi-Modal Learning. In ECCV

    2022

  7. [15]

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4510–4520

    2018

  8. [16]

    Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2016. Grad-CAM: Why did you say that? arXiv preprint arXiv:1611.07450 (2016)

    2016 arXiv

  9. [17]

    Aliza Subedi, Smriti Regmi, Nisha Regmi, Bhumi Bhusal, Ulas Bagci, and Debesh Jha. 2024. Classification of Endoscopy and Video Capsule Images using CNN- Transformer Model. In CaPTion@MICCAI. https://api.semanticscholar.org/ CorpusID:271909358

    2024

  10. [18]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  11. [19]

    Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. 2020. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition work...

    2020

  12. [20]

    Yuhao Zhang, Xiaoman Liu, Chunyuan Tao, et al . 2023. BioMedCLIP: Medi- cal Vision-Language Pretraining with Biomedical Knowledge. arXiv preprint arXiv:2303.09044 (2023)

    2023 arXiv

  13. [21]

    Yuhao Zhang, Chunyuan Tao, Shengjie Luan, et al. 2022. Contrastive learning of medical visual representations from paired images and text. In NeurIPS

    2022

  14. [22]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional Prompt Learning for Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16816–16825

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.