Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

ACM Multimedia Grand Challenge on ENT Endoscopy Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The paper introduces ENTRep as the first benchmark to jointly evaluate fine-grained ENT endoscopy classification with image-to-image and text-to-image retrieval under bilingual Vietnamese-English clinical supervision.

desk verdict A useful new ENT benchmark with a statistically underpowered retrieval leaderboard and some fixable documentation gaps; give it a serious, revision-minded peer review. read the letter →

arxiv 2508.04801 v1 pith:GTWN5J5G submitted 2025-08-06 cs.CV

classification cs.CV
keywords ENTendoscopymedicalimageclassificationimage-to-imageretrievaltext-to-imagevision-languagemodelsbenchmarkdatasetbilingualclinicaldescriptionsgrandchallenge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ENTRep, introduced as the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, is the paper's attempt to establish a first-of-its-kind benchmark: one expert-annotated dataset of ear, nose, throat, and vocal-fold endoscopic images that supports fine-grained anatomical classification, image-to-image retrieval of similar cases, and retrieval of images from concise English clinical descriptions, with the underlying narratives provided in both Vietnamese and English. The authors argue that standard benchmarks miss the distinctions clinicians actually make — left versus right sides, vocal cords open versus closed — and the case-retrieval workflow they depend on, and that recent vision-language advances make such a combined benchmark both timely and practical. If the benchmark is accepted, it gives the community a standardized, reproducible way to compare ENT endoscopy AI systems, and the reported results already locate the remaining difficulty: top classification entries sit near 95-96% accuracy, retrieval leaders reach roughly 88-95% Recall@1, and the text-to-image rankings shift markedly across test splits, showing that query phrasing, not just image content, drives performance.

What carries the argument

The load-bearing object is the ENTRep dataset and its evaluation protocol. The dataset is a set of endoscopic images from routine clinical practice, each carrying a seven-way anatomical label (Ear Right/Left, Nose Right/Left, Throat, VC-open/VC-closed), a normal/abnormal flag, and, for a subset, paired Vietnamese-English narrative descriptions. The three tasks — classification, image-to-image retrieval, and text-to-image retrieval — are evaluated server-side on public (645 images) and private (646 images) test splits that mirror the training distribution, with retrieval scored by Recall@1 and Mean Reciprocal Rank where relevance is defined as a ground-truth image match. This machinery conver

What would settle it

Take a random subsample of about 150 ENTRep images and have two independent ENT clinicians re-annotate the seven-way anatomical label and the normal/abnormal flag; if agreement falls well below the level implied by the top models' roughly 95% accuracy (say, below 95%), the ground truth is too noisy to support the claimed separation of models. A complementary check: paraphrase the 71 English text queries into a second set of 71 and re-run the leading text-to-image models — the paper's own public-to-private drops (top public team falling from 95.11% to 85.56% Recall@1) predict that rankings unde

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single expert-annotated resource can carry all three clinical workflows at once, and that a modest, class-imbalanced collection of endoscopy frames is enough to rank vision-language approaches in a way the authors can defend. Each ENTRep image is labeled with one of seven anatomical categories — Ear Right, Ear Left, Nose Right, Nose Left, Throat, VC-open, VC-closed — plus a normal/abnormal flag, and, where available, matched Vietnamese and English narrative descriptions; retrieval relevance is defined by ground-truth image pairings (141 training pairs, 139 per public and private test split, 71 text queries per split). The paper reports that t

Load-bearing premise

The benchmark's conclusions depend on the assumption that a single-institution image set — roughly 645 public and 646 private test images for classification, only 139 image pairs and 71 text queries per retrieval split, with expert annotations whose agreement is unreported — is large and reliable enough that the measured rankings and generalization gaps are stable and not noise; the paper itself, in its outlook, acknowledges that expanding beyond a single institution would in

Editorial extensions

If this is right

  • Classification on this benchmark is nearly saturated: the top entries cluster at 94-96% accuracy on both splits, so the paper's framework implies further progress must come from the specific error modes it names — left/right laterality and vocal-fold open/closed states — rather than from generic accuracy gains.
  • Image-to-image retrieval reaches 92-94% Recall@1 for the best systems, with the paper attributing early-precision gains to symmetric pairing losses, hard-negative mining, and anatomical gating of the gallery; these ingredients become the default recipe for clinical case retrieval.
  • Text-to-image retrieval is the least stable task: the leading public-set entry drops about 10 points in Recall@1 on the private set, which the paper reads as sensitivity to description phrasing and coverage, pointing toward synonym normalization and stronger language encoders as the next required step.
  • The small public-to-private degradation of the strongest models supports the paper's claim that the server-side protocol limits leaderboard overfitting, making ENTRep usable as a standardized comparison basis for future ENT endoscopy vision-language research.
  • Anatomy-aware augmentation — label-swapping flips for left/right classes, state-preserving flips for throat and vocal folds — recurs across winning submissions, so the paper's results imply that generic augmentation schemes are a measurable liability for fine-grained anatomical classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My read: with only 139 ground-truth pairs per retrieval split, Recall@1 is a high-variance statistic — a handful of pairs moving between ranks shifts the metric by about a point — so the tight leaderboard gaps (e.g., 92.09 vs 88.79 in private image retrieval) may not be robust; enlarging the paired-query pool or reporting confidence intervals would likely reorder the field. The paper does not addr
  • My read: the Vietnamese half of the supervision is an untapped test asset — the tasks only query in English, so one could directly probe cross-lingual retrieval by querying in Vietnamese and checking whether the same ground-truth images are retrieved; the paper neither promises nor tests this.
  • My read: the single-institution acquisition leaves device- and operator-induced domain shift unmeasured, so the benchmark's implicit promise that its rankings predict deployment would be best tested by replicating the three tasks on a second hospital's endoscopy archive of comparable size.
  • My read: the instability of text-to-image rankings across splits suggests a metric design choice — treating each 71-query set as a fixed evaluation rather than sampling over paraphrase variants — and formalizing stability under paraphrase as part of the task would make the text-to-image leaderboard more meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents ENTRep, the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, and positions it as the first benchmark combining fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual (Vietnamese/English) clinical supervision. The dataset consists of expert-annotated ENT endoscopic images from a single hospital, with labels for anatomical region and normal/abnormal status, plus paired bilingual descriptions. Three tasks are defined: seven-class image classification, image-to-image retrieval, and text-to-image retrieval. The paper specifies evaluation metrics (weighted accuracy/precision/recall/F1 for Task 1; Recall@1 and MRR for Tasks 2 and 3), describes public/private test splits and server-side scoring, and reports the top participating teams' results on both splits, followed by a qualitative discussion of what worked.

Significance. If adequately documented, ENTRep would fill a genuine gap: public ENT endoscopy benchmarks are scarce, and combining classification with bimodal retrieval under bilingual supervision is clinically useful. The organizers deserve credit for committing to a transparent evaluation protocol (server-side scoring, public/private splits, a unified output schema) and for reporting results from multiple independent teams, which the community can reuse. The dataset and challenge infrastructure are likely to lower the barrier for future work on ENT vision-language models. However, the current manuscript does not yet provide the statistical and methodological support needed for the 'standardized benchmark' claim: retrieval test sets are very small, no baselines are shown, and annotation reliability is unreported. The value of the resource is real, but the paper's stronger comparative claims are not yet supported.

major comments (4)
  1. [§4.1 and Tables 2–3] The retrieval evaluation is built on very small test sets: n=139 query images per split for Task 2 and n=71 text queries per split for Task 3. For a binary per-query outcome with n=71, the 95% Wilson interval for a Recall@1 of about 92.6% is roughly [82.7%, 98.1%]; for n=139 it is roughly ±5 percentage points. The top-team differences in Table 3 (e.g., 95.11 vs. 94.96 on public; 92.64 vs. 90.77 on private) and in Table 2 (e.g., 92.09 vs. 88.79 on private) are the same order as, or smaller than, this noise floor. Section 5 nonetheless interprets these differences as evidence of model strength and generalization gaps, and the private-test ranking determines challenge winners. Please report confidence intervals or bootstrap/McNemar-style significance tests, and explicitly state the statistical uncertainty around the leaderboard. Without this, the 'standardized comparison' claim is not suppo
  2. [§4.1] The training set size is never stated. The text says 'The public and private test sets closely mirror the class distributions and sample sizes of the training set (645 and 646 images, respectively),' which is ambiguous: are 645 and 646 the training sizes or the test sizes? The abstract and later text imply the test splits contain 645 and 646 images, but the exact composition of the training split is essential for reproducing the challenge and for interpreting the public-to-private generalization statements. Please provide explicit counts per split, per class, and per retrieval task, together with a data-access URL or a clear statement of where the dataset can be obtained.
  3. [§1 and §5 (Tables 1–3)] The introduction says the paper presents 'baseline results,' but no baselines are actually reported. Tables 1–3 list only challenge participant teams. Without a majority-class classifier, a random-retrieval baseline, a zero-shot CLIP/ViT baseline, or an organizer-run model trained on the provided training set, the reader cannot judge task difficulty or calibrate the 'tightly clustered' statements in Section 5. Adding at least simple baselines is necessary to support the benchmark's purpose of standardized comparison and to make the reported leaderboard differences interpretable.
  4. [§4.1] Annotation reliability is not reported. The paper states that 'Clinical experts annotate each image' and that descriptions were provided, but it does not specify the number of annotators, their experience levels, the adjudication procedure, or any inter-annotator agreement (e.g., Cohen's or Fleiss' kappa). Since classification includes fine-grained distinctions (laterality, vocal-fold open/closed) and since retrieval ground truth depends on expert-curated image pairs and text descriptions, agreement is a load-bearing quality indicator. Please report it, or explicitly justify why it is not needed for the challenge's conclusions.
minor comments (4)
  1. [§7] The conclusion contains a typo: 'clinically benchmark' should be 'clinical benchmark.'
  2. [§4.1] The text says the dataset is 'partitioned into three subsets, as illustrated in Fig. 1,' but Fig. 1 illustrates Task 1 classes. The partition diagram appears to be Fig. 4; please fix the cross-reference.
  3. [§4.2, Eq. (2)] The indicator function is typeset as '⊮[·]', which is nonstandard and easily confused with a negation symbol. Use \mathbb{1}[·] or bold 1. Also, N is used in Eq. (2) but defined only immediately before MRR; please make the definitions explicit for both R@1 and MRR.
  4. [§4.1] The word cloud in Fig. 5 is descriptive but not used in any quantitative analysis. Consider replacing it with a query-length histogram or a table of the most frequent terms, which would be more informative for retrieval tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a dataset and third-party challenge results, with no fitted-parameter or self-citation chain that reduces to its own inputs.

full rationale

The paper's central contribution is the ENTRep dataset and the organization of three benchmark tasks. It does not derive quantitative predictions from fitted parameters; instead, it reports leaderboard metrics submitted by independent challenge teams (Tables 1–3). The dataset statistics, evaluation metrics (Eqs. 1–2), and task definitions are standard benchmark specifications, not results obtained by fitting the data. The authors cite their own prior work (e.g., VoFoCD-related references [3,4,5,12]) only as background on the organizing team's experience and related medical-imaging efforts; these citations are not used to justify the central claim of being the first benchmark to jointly evaluate fine-grained ENT classification with intra-modal and cross-modal retrieval. The 'to the best of our knowledge' first-benchmark claim is stated directly rather than derived from a self-citation. No equation in the paper reproduces its inputs by construction, and no fitted parameter is relabeled as a prediction. Therefore, the paper is self-contained with respect to circularity concerns, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is a benchmark and dataset paper rather than a derivation, so no free parameters or invented entities appear. The main assumptions are about data quality, annotation reliability, and the validity of the retrieval relevance judgments.

assumptions (3)
  • domain assumption Expert annotations of anatomical region and normal/abnormal status are treated as ground truth without inter-annotator agreement or adjudication details.
    Section 4.1 states 'Clinical experts annotate each image' but provides no quality control metrics.
  • domain assumption The relevance criterion for retrieval tasks is a ground-truth match to the query's annotated image(s).
    Section 2.3 defines relevance as 'corresponds to the ground-truth image(s) annotated for the query'; this assumes the annotation covers all relevant images.
  • domain assumption Stratified sampling of test splits mirrors training class distributions, implying the private test is representative.
    Section 4.1 claims the splits mirror distributions, but no protocol details are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACM Multimedia Grand Challenge on ENT Endoscopy Analysis." pith.science (2026). https://pith.science/paper/GTWN5J5G

@misc{pith2026250804801,
  author       = {Pith},
  title        = {Pith review of: ACM Multimedia Grand Challenge on ENT Endoscopy Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTWN5J5G}},
  note         = {Machine review of arXiv:2508.04801}
}
read the original abstract

Automated analysis of endoscopic imagery is a critical yet underdeveloped component of ENT (ear, nose, and throat) care, hindered by variability in devices and operators, subtle and localized findings, and fine-grained distinctions such as laterality and vocal-fold state. In addition to classification, clinicians require reliable retrieval of similar cases, both visually and through concise textual descriptions. These capabilities are rarely supported by existing public benchmarks. To this end, we introduce ENTRep, the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, which integrates fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual (Vietnamese and English) clinical supervision. Specifically, the dataset comprises expert-annotated images, labeled for anatomical region and normal or abnormal status, and accompanied by dual-language narrative descriptions. In addition, we define three benchmark tasks, standardize the submission protocol, and evaluate performance on public and private test splits using server-side scoring. Moreover, we report results from the top-performing teams and provide an insight discussion.

Figures

Figures reproduced from arXiv: 2508.04801 by the authors.

Figure 2
Figure 2. Task 2 (Image-to-Image Retrieval): given an image, the system returns a ranked list of database images [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Task 3 (Text-to-Image Retrieval): given a clinical [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Overview of dataset partitioning and class distribution for the classification task. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Word cloud visualization of text queries. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification

    cs.CV 2025-08 conditional novelty 4.0 of 10

    A multi-task CLIP model with LoRA, multi-level CLS fusion, and spherical feature interpolation reports 95% accuracy and strong retrieval scores on the ENTRep endoscopy benchmark.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rohaid Ali, Oliver Y Tang, Ian D Connolly, Hael F Abdulrazeq, Fatima N Mirza, Rachel K Lim, Benjamin R Johnston, Michael W Groff, Theresa Williamson, Konstantina Svokos, et al. 2024. Demographic representation in 3 leading artificial intelligence text-to-image generators. JAMA surgery 159, 1 (2024), 87–95

  2. [2]

    Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, and Zhiwei Xiong. 2024. Bimcv-r: A landmark dataset for 3d ct text-image retrieval. In International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 124–134

  3. [3]

    Thao Thi Phuong Dao, Tuan-Luc Huynh, Minh-Khoi Pham, Trung-Nghia Le, Tan- Cong Nguyen, Quang-Thuc Nguyen, Bich Anh Tran, Boi Ngoc Van, Chanh Cong Ha, and Minh-Triet Tran. 2024. Improving Laryngoscopy Image Analysis Through Integration of Global Information and Local Features in VoFoCD Dataset.Journal of Imaging Informatics in Medicine 37, 6 (2024), 2794–2809

  4. [4]

    Thao Thi Phuong Dao, Tan-Cong Nguyen, Viet-Tham Huynh, Xuan-Hai Bui, Trung-Nghia Le, and Minh-Triet Tran. 2024. Artificial intelligence for laryn- goscopy in vocal fold diseases: a review of dataset, technology, and ethics.Machine Learning 113, 11 (2024), 8749–8767

  5. [5]

    Thao Thi Phuong Dao, Minh-Khoi Pham, Mai-Khiem Tran, Chanh Cong Ha, Boi Ngoc Van, Bich Anh Tran, and Minh-Triet Tran. 2024. Vision-Based Assistance for Vocal Fold Identification in Laryngoscopy with Knowledge Distillation. In MEDINFO 2023—The Future Is Accessible . IOS Press, 946–950

  6. [6]

    Emre Demir, Burak Numan Uğurlu, Gülay Aktar Uğurlu, and Gülçin Aydoğdu

  7. [7]

    Brian Hu, Bhavan Vasu, and Anthony Hoogs. 2022. X-mir: Explainable medical image retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 440–450

  8. [8]

    Viet-Tham Huynh, Trong-Thuan Nguyen, Thao Thi-Phuong Dao, Tam V Nguyen, and Minh-Triet Tran. 2024. DermAI: A Chatbot Assistant for Skin Lesion Di- agnosis Using Vision and Large Language Models. In Proceedings of the Asian Conference on Computer Vision . 287–301

Show all 15 references
  1. [9]

    Hai Le, Trong-Thuan Nguyen, and Minh-Triet Tran. 2024. MedGraph-RPE: Graph- Based Medical Segmentation Enhanced by Novel Relative Positioning Encod- ing. In International Symposium on Information and Communication Technology . Springer, 27–42

  2. [10]

    Trong-Thuan Nguyen, Tam V Nguyen, and Minh-Triet Tran. 2023. Collaborative consultation doctors model: Unifying cnn and vit for covid-19 diagnostic. IEEE Access 11 (2023), 95346–95357

  3. [11]

    Kim Anh Phung, Thuan Trong Nguyen, Nileshkumar Wangad, Samah Baraheem, Nguyen D Vo, and Khang Nguyen. 2022. Disease recognition in X-ray images with doctor consultation-inspired model. Journal of Imaging 8, 12 (2022), 323

  4. [12]

    Bich Anh Tran, Thao Thi Phuong Dao, Ho Dang Quy Dung, Ngoc Boi Van, Chanh Cong Ha, Nam Hoang Pham, Tu Cong Huyen Ton Nu Cam, Tan-Cong Nguyen, Minh-Khoi Pham, Mai-Khiem Tran, et al. 2023. Support of deep learn- ing to classify vocal fold images in flexible laryngoscopy. America...

  5. [13]

    Trong-Nghia Tran, Minh-Nhat Nguyen, Minh-Triet Tran, and Trong-Le Do. 2025. Enhancing Medical Chatbot Reliability: A Multi-Step Verification Approach to Prevent Hallucinations. In Proceedings of the 2nd Workshop on Security-Centric Strategies for Combating Information Disorder . 1–12

  6. [14]

    Qingwu Wu, Xinyue Wang, Guixian Liang, Xin Luo, Min Zhou, Huiyi Deng, Yana Zhang, Xuekun Huang, and Qintai Yang. 2023. Advances in image-based artificial intelligence in otorhinolaryngology–head and neck surgery: a systematic review. Otolaryngology–Head and Neck Surgery 169, 5...

  7. [2025]

    European Archives of Oto-Rhino-Laryngology (2025), 1–11

    Artificial intelligence in otorhinolaryngology: current trends and applica- tion areas. European Archives of Oto-Rhino-Laryngology (2025), 1–11

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.