{"id":"6d413dda-f83a-465f-9648-47358980298e","arxiv_id":"2508.04801","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ENTRep is a new ENT endoscopy benchmark combining seven-class anatomical classification, image-to-image retrieval, and text-to-image retrieval with bilingual clinical descriptions.","lead":"This paper introduces ENTRep, a benchmark dataset for ear, nose, and throat endoscopy images with tasks for classification and image retrieval. It reports results from top teams in the ACM Multimedia 2025 Grand Challenge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval test sets are so small that reported leaderboard differences are likely within statistical noise, undermining the benchmark's stated purpose of standardized comparison.","rationale":"The reader's weakest_assumption correctly identified the dataset's small scale and the missing annotation reliability metrics. However, the most load-bearing concern is more specific and technical: the retrieval test sets are too small to statistically support the benchmark's core function of providing a reliable standardized comparison. The paper's performance claims and winner determination rest on differences that are within the noise floor of the evaluation metrics. This does not invalidate the 'first benchmark' claim per se, but it weakens the usefulness of the benchmark for drawing conclusions about model superiority—one of the paper's primary contributions. The concrete bootstrap test would objectively settle whether the observed leaderboard gaps are distinguishable from random variation. Given the reader already set a CONDITIONAL verdict based on incompleteness, our concern reinforces that the benchmark needs either larger test sets or explicitly reported confidence intervals before its results can be taken as definitive. I recommend keeping the CONDITIONAL verdict; the paper should be accepted only if the authors add statistical validation or clearly caution that rankings are preliminary.","tokens_in":7416,"tokens_out":4881,"duration_ms":58330,"concrete_test":"Perform bootstrap resampling on the per-query retrieval results from the private test splits: for each team, resample the 71 (Task 3) or 139 (Task 2) query outcomes 10,000 times and compute 95% confidence intervals for Recall@1 and MRR. If the intervals of the top two teams overlap by more than a small fraction of the observed gap—or if the minimum detectable difference for n=71 at α=0.05 exceeds 5%—the ranking is not statistically robust. This would demonstrate whether the reported leaderboard differences are meaningful or noise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim asserts that ENTRep enables standardized comparison for fine-grained ENT classification and retrieval. However, the retrieval evaluation is built on extremely small test sets: Task 2 has only 139 query images per split (Section 4.1), and Task 3 has only 71 text queries per split (Section 4.1). For a binary success metric like Recall@1 with n=71, the 95% Wilson interval width is roughly ±5% (e.g., for p=0.95, interval ≈ [0.87, 0.98]). The reported gaps between top teams in Tables 2 and 3 are often 1–2% or less (e.g., 92.09 vs. 88.79 on private Task 2; 95.11 vs. 94.96 on public Task 3). These differences are far smaller than the noise floor, meaning a handful of query outcomes could reorder the leaderboard. The paper interprets these small differences as evidence of relative model strength (Section 5) and draws conclusions about generalization gaps (e.g., H3N1's drop to 85.56 on private Task 3) without any confidence intervals or statistical tests. This directly weakens the benchmark's ability to 'enable standardized comparison' and makes the private-test final ranking—which determines the challenge winners—statistically fragile. The issue is compounded by the near-one-to-one retrieval design (Section 4.1), where each query has essentially one relevant image, so the metric is highly sensitive to annotation noise and the tiny positive set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ENTRep, the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, and positions it as the first benchmark combining fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual (Vietnamese/English) clinical supervision. The dataset consists of expert-annotated ENT endoscopic images from a single hospital, with labels for anatomical region and normal/abnormal status, plus paired bilingual descriptions. Three tasks are defined: seven-class image classification, image-to-image retrieval, and text-to-image retrieval. The paper specifies evaluation metrics (weighted accuracy/precision/recall/F1 for Task 1; Recall@1 and MRR for Tasks 2 and 3), describes public/private test splits and server-side scoring, and reports the top participating teams' results on both splits, followed by a qualitative discussion of what worked.","tokens_in":7755,"tokens_out":3910,"duration_ms":47597,"significance":"If adequately documented, ENTRep would fill a genuine gap: public ENT endoscopy benchmarks are scarce, and combining classification with bimodal retrieval under bilingual supervision is clinically useful. The organizers deserve credit for committing to a transparent evaluation protocol (server-side scoring, public/private splits, a unified output schema) and for reporting results from multiple independent teams, which the community can reuse. The dataset and challenge infrastructure are likely to lower the barrier for future work on ENT vision-language models. However, the current manuscript does not yet provide the statistical and methodological support needed for the 'standardized benchmark' claim: retrieval test sets are very small, no baselines are shown, and annotation reliability is unreported. The value of the resource is real, but the paper's stronger comparative claims are not yet supported.","major_comments":[{"comment":"The retrieval evaluation is built on very small test sets: n=139 query images per split for Task 2 and n=71 text queries per split for Task 3. For a binary per-query outcome with n=71, the 95% Wilson interval for a Recall@1 of about 92.6% is roughly [82.7%, 98.1%]; for n=139 it is roughly ±5 percentage points. The top-team differences in Table 3 (e.g., 95.11 vs. 94.96 on public; 92.64 vs. 90.77 on private) and in Table 2 (e.g., 92.09 vs. 88.79 on private) are the same order as, or smaller than, this noise floor. Section 5 nonetheless interprets these differences as evidence of model strength and generalization gaps, and the private-test ranking determines challenge winners. Please report confidence intervals or bootstrap/McNemar-style significance tests, and explicitly state the statistical uncertainty around the leaderboard. Without this, the 'standardized comparison' claim is not suppo","section":"§4.1 and Tables 2–3"},{"comment":"The training set size is never stated. The text says 'The public and private test sets closely mirror the class distributions and sample sizes of the training set (645 and 646 images, respectively),' which is ambiguous: are 645 and 646 the training sizes or the test sizes? The abstract and later text imply the test splits contain 645 and 646 images, but the exact composition of the training split is essential for reproducing the challenge and for interpreting the public-to-private generalization statements. Please provide explicit counts per split, per class, and per retrieval task, together with a data-access URL or a clear statement of where the dataset can be obtained.","section":"§4.1"},{"comment":"The introduction says the paper presents 'baseline results,' but no baselines are actually reported. Tables 1–3 list only challenge participant teams. Without a majority-class classifier, a random-retrieval baseline, a zero-shot CLIP/ViT baseline, or an organizer-run model trained on the provided training set, the reader cannot judge task difficulty or calibrate the 'tightly clustered' statements in Section 5. Adding at least simple baselines is necessary to support the benchmark's purpose of standardized comparison and to make the reported leaderboard differences interpretable.","section":"§1 and §5 (Tables 1–3)"},{"comment":"Annotation reliability is not reported. The paper states that 'Clinical experts annotate each image' and that descriptions were provided, but it does not specify the number of annotators, their experience levels, the adjudication procedure, or any inter-annotator agreement (e.g., Cohen's or Fleiss' kappa). Since classification includes fine-grained distinctions (laterality, vocal-fold open/closed) and since retrieval ground truth depends on expert-curated image pairs and text descriptions, agreement is a load-bearing quality indicator. Please report it, or explicitly justify why it is not needed for the challenge's conclusions.","section":"§4.1"}],"minor_comments":[{"comment":"The conclusion contains a typo: 'clinically benchmark' should be 'clinical benchmark.'","section":"§7"},{"comment":"The text says the dataset is 'partitioned into three subsets, as illustrated in Fig. 1,' but Fig. 1 illustrates Task 1 classes. The partition diagram appears to be Fig. 4; please fix the cross-reference.","section":"§4.1"},{"comment":"The indicator function is typeset as '⊮[·]', which is nonstandard and easily confused with a negation symbol. Use \\mathbb{1}[·] or bold 1. Also, N is used in Eq. (2) but defined only immediately before MRR; please make the definitions explicit for both R@1 and MRR.","section":"§4.2, Eq. (2)"},{"comment":"The word cloud in Fig. 5 is descriptive but not used in any quantitative analysis. Consider replacing it with a query-length histogram or a table of the most frequent terms, which would be more informative for retrieval tasks.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward, honest challenge description that gives the medical imaging community a new resource: an ENT endoscopy dataset with a clean three-task setup (classification, image-to-image retrieval, text-to-image retrieval) and bilingual clinical text. The task definitions are precise, the metrics are standard, and the server-side scoring is good practice. The authors deserve credit for organizing the challenge and reporting team results transparently. If the dataset is released as described, it fills a real gap and will likely be cited.\n\nThe main problem is that the retrieval leaderboards are statistically underpowered. Task 3 has only 71 queries per split; Task 2 has 139. With binary metrics like Recall@1, a 1-2% gap between teams is within the noise floor. The paper not only ranks teams on these differences but interprets single numbers -- e.g., H3N1's drop from 95.11 to 85.56 on Task 3 -- as evidence about method sensitivity. That particular drop is large enough to be real, but most cross-team gaps in Tables 2 and 3 are not. The stress-test note is correct: the leaderboard rankings are fragile, which directly undercuts the benchmark's stated purpose of \"standardized comparison.\" The authors should add bootstrap confidence intervals or at least acknowledge the noise floor and soften ranking language.\n\nThere are also documentation gaps that are easy to fix. The training set size is never stated; the text confusingly lists 645 and 646 images under \"training set.\" No baselines appear despite the intro promising them, so readers cannot judge whether the teams' sophisticated pipelines beat a simple nearest-neighbor or CLIP-zero-shot baseline. There is no inter-annotator agreement for the expert labels, which matters for a medical benchmark. And the \"first benchmark\" claim is asserted rather than demonstrated by comparing against prior ENT datasets. The citation pattern is fine, though; the authors build on their own VoFoCD work appropriately.\n\nNone of this is fatal. The paper is a challenge description, not a methods paper, and its central contribution -- the dataset and task suite -- is plausible and likely useful. I agree with the reader's conditional take, but would emphasize the noise issue as the sharpest concern. A serious referee should engage, with the expectation that the authors add baselines, confidence intervals, state the training split size, and temper claims about leaderboard ranking.","headline":"A useful new ENT benchmark with a statistically underpowered retrieval leaderboard and some fixable documentation gaps; give it a serious, revision-minded peer review.","tokens_in":8239,"tokens_out":3270,"would_cite":true,"duration_ms":35560,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces ENTRep as the first benchmark to jointly evaluate fine-grained ENT endoscopy classification with image-to-image and text-to-image retrieval under bilingual Vietnamese-English clinical supervision.","keywords":["ENT endoscopy","medical image classification","image-to-image retrieval","text-to-image retrieval","vision-language models","benchmark dataset","bilingual clinical descriptions","grand challenge"],"falsifier":"Take a random subsample of about 150 ENTRep images and have two independent ENT clinicians re-annotate the seven-way anatomical label and the normal/abnormal flag; if agreement falls well below the level implied by the top models' roughly 95% accuracy (say, below 95%), the ground truth is too noisy to support the claimed separation of models. A complementary check: paraphrase the 71 English text queries into a second set of 71 and re-run the leading text-to-image models — the paper's own public-to-private drops (top public team falling from 95.11% to 85.56% Recall@1) predict that rankings unde","tokens_in":7358,"feed_emoji":"🩺","tokens_out":13573,"duration_ms":143196,"temperature":0.7,"pith_summary":"ENTRep, introduced as the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, is the paper's attempt to establish a first-of-its-kind benchmark: one expert-annotated dataset of ear, nose, throat, and vocal-fold endoscopic images that supports fine-grained anatomical classification, image-to-image retrieval of similar cases, and retrieval of images from concise English clinical descriptions, with the underlying narratives provided in both Vietnamese and English. The authors argue that standard benchmarks miss the distinctions clinicians actually make — left versus right sides, vocal cords open versus closed — and the case-retrieval workflow they depend on, and that recent vision-language advances make such a combined benchmark both timely and practical. If the benchmark is accepted, it gives the community a standardized, reproducible way to compare ENT endoscopy AI systems, and the reported results already locate the remaining difficulty: top classification entries sit near 95-96% accuracy, retrieval leaders reach roughly 88-95% Recall@1, and the text-to-image rankings shift markedly across test splits, showing that query phrasing, not just image content, drives performance.","feed_headline":"First ENT endoscopy benchmark runs classification plus retrieval","feed_subtitle":"Seven region labels, bilingual queries, public and private tests: top models already reach ~96% accuracy.","key_machinery":"The load-bearing object is the ENTRep dataset and its evaluation protocol. The dataset is a set of endoscopic images from routine clinical practice, each carrying a seven-way anatomical label (Ear Right/Left, Nose Right/Left, Throat, VC-open/VC-closed), a normal/abnormal flag, and, for a subset, paired Vietnamese-English narrative descriptions. The three tasks — classification, image-to-image retrieval, and text-to-image retrieval — are evaluated server-side on public (645 images) and private (646 images) test splits that mirror the training distribution, with retrieval scored by Recall@1 and Mean Reciprocal Rank where relevance is defined as a ground-truth image match. This machinery conver","core_discovery":"On the paper's own terms, the discovery is that a single expert-annotated resource can carry all three clinical workflows at once, and that a modest, class-imbalanced collection of endoscopy frames is enough to rank vision-language approaches in a way the authors can defend. Each ENTRep image is labeled with one of seven anatomical categories — Ear Right, Ear Left, Nose Right, Nose Left, Throat, VC-open, VC-closed — plus a normal/abnormal flag, and, where available, matched Vietnamese and English narrative descriptions; retrieval relevance is defined by ground-truth image pairings (141 training pairs, 139 per public and private test split, 71 text queries per split). The paper reports that t","pith_inferences":["My read: with only 139 ground-truth pairs per retrieval split, Recall@1 is a high-variance statistic — a handful of pairs moving between ranks shifts the metric by about a point — so the tight leaderboard gaps (e.g., 92.09 vs 88.79 in private image retrieval) may not be robust; enlarging the paired-query pool or reporting confidence intervals would likely reorder the field. The paper does not addr","My read: the Vietnamese half of the supervision is an untapped test asset — the tasks only query in English, so one could directly probe cross-lingual retrieval by querying in Vietnamese and checking whether the same ground-truth images are retrieved; the paper neither promises nor tests this.","My read: the single-institution acquisition leaves device- and operator-induced domain shift unmeasured, so the benchmark's implicit promise that its rankings predict deployment would be best tested by replicating the three tasks on a second hospital's endoscopy archive of comparable size.","My read: the instability of text-to-image rankings across splits suggests a metric design choice — treating each 71-query set as a fixed evaluation rather than sampling over paraphrase variants — and formalizing stability under paraphrase as part of the task would make the text-to-image leaderboard more meaningful."],"forward_implications":["Classification on this benchmark is nearly saturated: the top entries cluster at 94-96% accuracy on both splits, so the paper's framework implies further progress must come from the specific error modes it names — left/right laterality and vocal-fold open/closed states — rather than from generic accuracy gains.","Image-to-image retrieval reaches 92-94% Recall@1 for the best systems, with the paper attributing early-precision gains to symmetric pairing losses, hard-negative mining, and anatomical gating of the gallery; these ingredients become the default recipe for clinical case retrieval.","Text-to-image retrieval is the least stable task: the leading public-set entry drops about 10 points in Recall@1 on the private set, which the paper reads as sensitivity to description phrasing and coverage, pointing toward synonym normalization and stronger language encoders as the next required step.","The small public-to-private degradation of the strongest models supports the paper's claim that the server-side protocol limits leaderboard overfitting, making ENTRep usable as a standardized comparison basis for future ENT endoscopy vision-language research.","Anatomy-aware augmentation — label-swapping flips for left/right classes, state-preserving flips for throat and vocal folds — recurs across winning submissions, so the paper's results imply that generic augmentation schemes are a measurable liability for fine-grained anatomical classification."],"supporting_citations":[{"why":"Supplies the closest prior benchmark for medical text-image retrieval (3D CT, BIMCV-R), establishing the gap ENTRep extends to ENT endoscopy.","marker":"[2]"},{"why":"Review of AI in otorhinolaryngology that grounds the claim that ENT endoscopy is clinically significant yet underexplored.","marker":"[6]"},{"why":"Prior explainable medical image retrieval work (X-MIR) cited as evidence that vision-language alignment enables versatile retrieval systems.","marker":"[7]"},{"why":"Systematic review of image-based AI in otolaryngology-head and neck surgery, the domain baseline the benchmark positions itself against.","marker":"[14]"},{"why":"The organizers' earlier VoFoCD laryngoscopy dataset and analysis, informing the ENT-specific annotation and task framing used in ENTRep.","marker":"[3]"}],"fun_headline_variants":["ENT endoscopy benchmark debuts with classification and retrieval","Seven anatomical labels, bilingual queries: new ENT endoscopy challenge","ENTRep: one dataset powers three clinical workflows for ENT endoscopy","First ENT endoscopy AI benchmark combines fine-grained labels and retrieval","Bilingual retrieval joins classification in first ENT endoscopy benchmark"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The benchmark's conclusions depend on the assumption that a single-institution image set — roughly 645 public and 646 private test images for classification, only 139 image pairs and 71 text queries per retrieval split, with expert annotations whose agreement is unreported — is large and reliable enough that the measured rankings and generalization gaps are stable and not noise; the paper itself, in its outlook, acknowledges that expanding beyond a single institution would in","fun_headline_variants_meta":{"raw":{"variants":["ENT endoscopy benchmark debuts with classification and retrieval","Seven anatomical labels, bilingual queries: new ENT endoscopy challenge","ENTRep: one dataset powers three clinical workflows for ENT endoscopy","First ENT endoscopy AI benchmark combines fine-grained labels and retrieval","Bilingual retrieval joins classification in first ENT endoscopy benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2481,"prompt_tokens":703,"completion_tokens":1778,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1694}},"tokens_in":447,"tokens_out":1778,"duration_ms":18927,"temperature":1.0,"reasoning_tokens":1694,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:45:40.687238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subsample of about 150 ENTRep images and have two independent ENT clinicians re-annotate the seven-way anatomical label and the normal/abnormal flag; if agreement falls well below the level implied by the top models' roughly 95% accuracy (say, below 95%), the ground truth is too noisy to support the claimed separation of models. A complementary check: paraphrase the 71 English text queries into a second set of 71 and re-run the leading text-to-image models — the paper's own public-to-private drops (top public team falling from 95.11% to 85.56% Recall@1) predict that rankings unde","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the closest prior benchmark for medical text-image retrieval (3D CT, BIMCV-R), establishing the gap ENTRep extends to ENT endoscopy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Review of AI in otorhinolaryngology that grounds the claim that ENT endoscopy is clinically significant yet underexplored."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior explainable medical image retrieval work (X-MIR) cited as evidence that vision-language alignment enables versatile retrieval systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Systematic review of image-based AI in otolaryngology-head and neck surgery, the domain baseline the benchmark positions itself against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The organizers' earlier VoFoCD laryngoscopy dataset and analysis, informing the ENT-specific annotation and task framing used in ENTRep."}],"review_version":1}