Pith. sign in

REVIEW 3 major objections 6 minor 26 references

Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt wording—especially naming the exact organ—decides how well vision-language models diagnose cancer from whole-slide images, and the domain-trained CONCH beats a far larger model when the prompt gives precise anatomical context.

desk verdict Useful probe of prompt sensitivity in pathology VLMs, but the AUC numbers rest on an unspecified patch-to-slide pooling and one model that may not support ROC at all. read the letter →

arxiv 2505.00134 v1 pith:CEEMVGO7 submitted 2025-04-30 cs.CV

Vasudev Sharma , Ahmed Alagha , Abdelhakim Khellaf , Vincent Quoc-Huy Trinh , Mahdi S. Hosseini This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelscomputationalpathologypromptengineeringzero-shotclassificationhistopathologywholeslideimagesanatomicalprecisioncancerinvasiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that how a prompt is worded—particularly whether it names the exact organ and tissue—changes whether off-the-shelf vision-language models can detect invasive cancer and dysplasia in whole slide images. On an in-house set of 3,507 digestive-tract slides, the authors compare three models (Quilt-Net, Quilt-LLAVA, and CONCH) without any fine-tuning. They find that CONCH, a model trained specifically on histopathology image-text pairs, reaches the highest accuracy when the prompt gives precise anatomical context, and that all models lose accuracy when anatomical detail is removed. They also argue that a much larger model (Quilt-LLAVA, roughly 7 billion parameters) does not beat a smaller domain-trained one (CONCH, roughly 200 million), so domain alignment matters more than scale. The practical stakes are that prompt design becomes a cheap, parameter-free lever for improving AI-assisted pathology diagnosis.

What carries the argument

The machinery is the prompt-engineering framework itself: a set of nine prompt templates spanning four dimensions—detail specificity (LOW/MEDIUM/HIGH), anatomical precision (LOW/MEDIUM/HIGH), instructional framing (MINIMAL/TASK/EXPERT), and output constraints (EXPLICIT/IMPLICIT)—plus the three pretrained VLMs run zero-shot. The templates P1–P9 and D1–D3 are the experimental instruments: by holding three dimensions fixed and varying one, the paper isolates what in the linguistic input moves the AUC. The load-bearing comparison is prompt P5 (high anatomical precision, medium specificity) against P6–P8 (progressively vaguer anatomy), which directly tests the anatomical-context hypothesis, and the model comparison CONCH versus Quilt-LLAVA, which tests scale-versus-alignment.

What would settle it

Recompute slide-level AUCs after defining an explicit patch-to-slide pooling rule (e.g., mean or max of patch probabilities per slide) and re-rank the three models; if the CONCH-versus-Quilt-LLAVA ordering or the anatomical-precision gradient changes under a different pooling rule, the conclusion that prompt anatomy drives performance would not generalize.

Watch

Extended reading notes

Core claim

The central claim is that zero-shot performance of vision-language models on pathology diagnostics is highly sensitive to prompt composition, and that the best prompt is one with intermediate information density, high anatomical precision, and explicit output constraints. In the ablative study, prompt P5—which names the organ (e.g., colon wall, lymph node) and asks for a single-word label—produced the highest AUC for all three models, with CONCH reaching 0.935 for invasive cancer classification. When the anatomical reference was weakened (prompts P6–P8, moving from organ to 'gastrointestinal system' to 'digestive system' to none), performance consistently dropped. The paper further shows that near-synonymous medical terms ('dysplasia', 'atypia', 'precancerous') shift which prompt works best for each model, and that expert-role framing ('You are an expert pathologist') hurts or is neutral in two of the three models. The conclusion is that domain-specific pretraining and careful prompt anatomy, not parameter count, determine diagnostic accuracy, and the authors position this as a starting guideline for prompt engineering in computational pathology.

Load-bearing premise

The paper assumes that every 512×512 patch cut from a slide carries the slide's own diagnosis, and that the way patch scores are turned into a slide-level ROC curve is valid—but it never says how that pooling is done, so a patch of normal tissue next to a tumor is counted as tumor.

Editorial extensions

If this is right

  • Zero-shot pathology screening can be improved with no retraining by naming the exact organ in the prompt; a single phrase change is worth more than swapping in a larger model.
  • Model rankings in medical VLM benchmarks are unstable across prompt wordings, so evaluations should report a family of prompts rather than a single template.
  • For clinical deployment, prompts with explicit output constraints and medium information density are a sensible default starting point.
  • The choice of terminology for a condition (dysplasia vs atypia vs precancerous) should be matched to the model, since the best synonym differs by model.
  • Attention-map behavior tracks diagnostic quality: the model with the highest AUC also highlights invasive cancer and precursor lesions in a way a pathologist recognizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be a broader grid over organ taxonomies (adding laterality, biopsy site, or patient age); the benefit of anatomical precision may be an inverted-U rather than monotonic, with diminishing or negative returns beyond a moderate level.
  • The patch-labeling shortcut (every patch inherits the slide label) is common, but re-analysis with attention-based multiple-instance pooling could narrow the CONCH-versus-Quilt-LLAVA gap, since Quilt-LLAVA's patch-level scores are binary.
  • The near-synonym result suggests these models rely on lexical associations from pretraining rather than reasoning about underlying pathology; a probe with adversarial synonyms (e.g., 'malignant' vs 'cancerous') could test how systematic the sensitivity is.
  • Clinically, the finding implies prompt design should be part of regulatory evaluation of AI pathology tools, since a favorable AUC could in part be a prompt artifact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript reports a systematic zero-shot evaluation of three vision-language models (Quilt-Net, Quilt-LLAVA, CONCH) on a large in-house digestive pathology dataset of 3,507 WSIs. The authors design nine invasiveness prompts and three dysplasia prompts that vary detail specificity, anatomical precision, instructional framing, and output constraints; they report ROC/AUC for invasiveness and dysplasia, per-tissue AUC values, a magnification-level analysis, and pathologist-reviewed attention heatmaps. The central claims are that prompt engineering significantly changes zero-shot diagnostic performance, that CONCH is the best model when given precise anatomical references, and that reducing anatomical precision consistently degrades performance.

Significance. The study has several strengths: it evaluates three publicly available pathology VLMs, uses a clinically realistic large dataset spanning seven digestive tissue types, provides a structured ablative prompt framework, and includes qualitative attention-map validation by a board-certified pathologist. If the evaluation protocol were fully specified and statistically grounded, the paper would offer practically useful evidence on prompt sensitivity in zero-shot pathology VLMs and on the limited role of model scale relative to domain alignment. However, the central quantitative claims are not currently verifiable because the unit of analysis is ambiguous, and the paper does not report uncertainty estimates. The significance of the reported model and prompt ordering therefore cannot yet be assessed.

major comments (3)
  1. [§4 and §5.1 (Figs. 3–4)] The manuscript does not specify how patch-level outputs are pooled into the slide-level ROC/AUC curves. Section 4 states that WSIs are tiled into 512×512 patches at 5× with a sliding window and 0% overlap, while Section 5.1 reports ROC curves for invasiveness; each WSI has a single slide-level label (Table 1). If every patch inherits the slide label, then non-invasive tissue within an invasive WSI (e.g., normal mucosa, stroma, background) is treated as invasive, and the reported AUC reflects patch-count weighting and slide size rather than slide-level diagnostic accuracy. This is load-bearing for the headline ordering P5 > P6 > P7 > P8 and the average AUC ranking (CONCH 0.876 > Quilt-LLAVA 0.753 > Quilt-Net 0.666). Please specify the exact aggregation rule (e.g., mean or max patch score per slide, or a multiple-instance model) and report slide-level ROC/AUC with the corresponding effective sample size.
  2. [§5.2 vs. §5.1] There is a direct contradiction between the attention-map protocol and the ROC analysis for Quilt-LLAVA. Section 5.2 states that Quilt-LLAVA 'does not inherently produce continuous confidence scores; therefore, binary labels are used' for heatmaps, yet Section 5.1 and Figure 3(b) report ROC curves and an AUC of 0.807 for Quilt-LLAVA under P5. A ROC curve requires a continuous score or at least a ranking; with binary patch labels the ROC degenerates to a single operating point and the AUC becomes a fraction of positive pairs, not a threshold-averaged measure. Please clarify how the ROC was constructed for Quilt-LLAVA, or restrict its evaluation to the appropriate metric.
  3. [§5.1, Table 3, Figs. 3–4] No error bars, confidence intervals, or significance tests are reported anywhere in the results. The dataset contains 3,507 slides, but if the analysis is per-patch the effective sample size is inflated by thousands of patches per slide; if per-slide pooling is used, the variance across slides should be reported. The per-tissue AUCs in Table 3 are especially affected, as classes such as AW (n=108), MCW (n=45), and GJ (n=40) are small, yet differences such as GJ=0.94 vs AW=0.59 are interpreted without uncertainty. The differences used to support claims such as the degrading effect of anatomical precision and the average AUC ordering may be within sampling variability. Please provide bootstrap or other uncertainty estimates at the slide level and, where appropriate, statistical tests for the prompt and model comparisons.
minor comments (6)
  1. [§5.1] There is a typo: 'Quitl-LLAVA' should be 'Quilt-LLAVA' in the sentence reporting the P5 AUC comparison.
  2. [Title and §2] The title mentions 'Efficient Prompt Design,' but the paper reports no efficiency metrics such as inference time, GPU hours, or token counts; either add such measurements or adjust the title and framing.
  3. [Fig. 6] The magnification-level analysis does not state which prompt template is used, nor does it provide confidence intervals or the number of slides per magnification level; please specify these details.
  4. [§4] The prompt framework is described as 'based on information theory and clinical communication principles,' but no formal derivation or supporting reference is provided; a brief justification or citation would strengthen the presentation.
  5. [§3 and §5.1] The paper does not mention whether multiple WSIs can originate from the same patient; if they can, slide-level clustering should be accounted for in the statistical analysis.
  6. [§4 and §5] The prompt templates and the best-prompt conclusions were developed and evaluated on the same dataset; a validation split or an explicit acknowledgment of this limitation would help temper the generalizability claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all reported comparisons are direct zero-shot measurements on frozen external vision-language models, with no fitted parameters or self-referential derivations.

full rationale

I walked the paper's derivation chain and found no step in which a prediction reduces to its own input by construction. The study measures the zero-shot performance of three externally pretrained VLMs on an in-house WSI dataset using hand-specified prompt templates; the AUC values in Section 5.1 and Table 3 are direct empirical outputs, not quantities fitted to the data and then renamed as predictions. The central claim that CONCH performs best with precise anatomical prompts is a summary of measured results, and the claim that performance degrades when anatomical precision is reduced is likewise an observed ordering across prompts P5-P8, not a derivation from an assumption that already contains that ordering. The prompt framework is an experimental design choice, and the fact that it was evaluated on the same dataset is a generalization and selection concern, not definitional circularity; no parameter is fitted to a subset and then used to predict a closely related quantity. The only self-citation, reference [11], is a survey citation used for background and is not load-bearing for any claim. The missing patch-to-slide pooling description is a correctness and interpretability risk regarding what the reported ROC/AUC curves mean, but it does not make any claimed result equivalent to its input by definition. Accordingly, no circular step meets the evidentiary standard required here, and the paper is self-contained as an empirical benchmark study.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper depends on pathologist-provided slide labels as ground truth, on the implicit assumption that patch-level scores can be pooled with WSI-level labels to form slide-level ROC curves, and on hand-chosen experimental settings. No free parameters are fitted to the data; the central results are measurements, not derivations.

free parameters (3)
  • Magnification level = 5x
    Patch extraction is performed at 5x magnification (Section 4); this is a hand-chosen setting, and Figure 6 shows performance varies with magnification.
  • Patch size = 512x512 pixels
    Sliding-window patch size chosen without stated justification (Section 4).
  • Quilt-LLAVA decoding temperature = 0.1
    Inference temperature set to 0.1 to reduce hallucination (Section 5); results may depend on this choice.
assumptions (4)
  • domain assumption Slide-level pathologist annotations are accurate ground truth for invasiveness and dysplasia.
    All evaluations use these labels (Section 3, Table 1); no validation or inter-observer analysis is reported.
  • domain assumption Patch-level predictions with slide-level labels can be pooled to construct slide-level ROC/AUC.
    This is assumed implicitly; Section 4 describes patch extraction but Section 5.1 reports ROC curves without specifying patch-to-slide aggregation.
  • ad hoc to paper The in-house CNN for tissue/background segmentation correctly identifies tissue patches.
    Section 5.2 mentions an in-house CNN but gives no details, training data, or validation.
  • domain assumption Zero-shot inference with pretrained weights is a valid proxy for diagnostic capability.
    The authors deliberately omit fine-tuning (Section 4) and interpret resulting AUCs as measures of model-prompt performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design." pith.science (2026). https://pith.science/paper/CEEMVGO7

@misc{pith2026250500134,
  author       = {Pith},
  title        = {Pith review of: Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEEMVGO7}},
  note         = {Machine review of arXiv:2505.00134}
}
read the original abstract

Vision-language models (VLMs) have gained significant attention in computational pathology due to their multimodal learning capabilities that enhance big-data analytics of giga-pixel whole slide image (WSI). However, their sensitivity to large-scale clinical data, task formulations, and prompt design remains an open question, particularly in terms of diagnostic accuracy. In this paper, we present a systematic investigation and analysis of three state of the art VLMs for histopathology, namely Quilt-Net, Quilt-LLAVA, and CONCH, on an in-house digestive pathology dataset comprising 3,507 WSIs, each in giga-pixel form, across distinct tissue types. Through a structured ablative study on cancer invasiveness and dysplasia status, we develop a comprehensive prompt engineering framework that systematically varies domain specificity, anatomical precision, instructional framing, and output constraints. Our findings demonstrate that prompt engineering significantly impacts model performance, with the CONCH model achieving the highest accuracy when provided with precise anatomical references. Additionally, we identify the critical importance of anatomical context in histopathological image analysis, as performance consistently degraded when reducing anatomical precision. We also show that model complexity alone does not guarantee superior performance, as effective domain alignment and domain-specific training are critical. These results establish foundational guidelines for prompt engineering in computational pathology and highlight the potential of VLMs to enhance diagnostic accuracy when properly instructed with domain-appropriate prompts.

Figures

Figures reproduced from arXiv: 2505.00134 by the authors.

Figure 1
Figure 1. High level overview of the inference process for the three VLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Sample images from the in-house dataset detected in 2.08% (n = 73). This imbalance reflects real world clinical scenarios where pathological findings often represent a small subset of examined tissue. Importantly, the distribution of dysplasia varies considerably across tissue types. Dysplastic changes are predominantly observed in CW specimens, where 18.20% of CW specimens exhibit some degree of dysplasia (13.08% h… view at source ↗
Figure 3
Figure 3. ROC curves comparing the performance of three vision-language models: [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison of VLM models in terms of ROC AUC. CONCH [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of VLM models and prompts in terms of ROC [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Average ROC curves comparing model performance at different magnifi [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Comparison of different models across multiple WSI samples [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)

    2021

  2. [2]

    arXiv preprint arXiv:2408.14496 (2024)

    Chanda, D., Aryal, M., Soltani, N.Y., Ganji, M.: A new era in computa- tional pathology: A survey on foundation and vision-language models. arXiv preprint arXiv:2408.14496 (2024)

    arXiv 2024

  3. [3]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, R.J., Chen, C., Li, Y., Chen, T.Y., Trister, A.D., Krishnan, R.G., Mahmood, F.: Scaling vision transformers to gigapixel images via hierar- chical self-supervised learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16144–16155 (2022)

    work page 2022

  4. [4]

    arXiv preprint arXiv:2308.15474 (2023)

    Chen, R.J., et al.: A general-purpose self-supervised model for computa- tional pathology. arXiv preprint arXiv:2308.15474 (2023)

    arXiv 2023

  5. [5]

    Nature Communications16(1), 1239 (2025)

    Clusmann, J., Ferber, D., Wiest, I.C., Schneider, C.V., Brinker, T.J., Foer- sch,S.,Truhn,D.,Kather,J.N.:Promptinjectionattacksonvisionlanguage models in oncology. Nature Communications16(1), 1239 (2025)

    work page 2025

  6. [6]

    Laboratory Investigation101(4), 412–422 (2021)

    Cui, M., Zhang, D.Y.: Artificial intelligence and computational pathology. Laboratory Investigation101(4), 412–422 (2021)

    work page 2021

  7. [7]

    In: 2009 IEEE conference on com- puter vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on com- puter vision and pattern recognition. pp. 248–255. Ieee (2009)

    work page 2009

  8. [8]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2020

Show all 26 references
  1. [9]

    Advances in Neural Information Processing Sys- tems 37, 126725–126766 (2024)

    Du, Y., Sun, W., Snoek, C.: Ipo: Interpretable prompt optimization for vision-language models. Advances in Neural Information Processing Sys- tems 37, 126725–126766 (2024)

    2024

  2. [10]

    arXiv preprint arXiv:2304.08479 (2023)

    Gu, J., Beirami, A., Wang, X., Beutel, A., Torr, P., Qin, Y.: Towards robust prompts on vision-language models. arXiv preprint arXiv:2304.08479 (2023)

    2023 arXiv

  3. [11]

    Journal of Pathology Informatics15, 100357 (2024)

    Hosseini, M.S., et al.: Computational pathology: a survey review and the way forward. Journal of Pathology Informatics15, 100357 (2024)

    2024

  4. [12]

    Ikezogwo, W., et al.: Quilt-1m: One million image-text pairs for histopathol- ogy. Adv. in neural information processing systems36, 37995–38017 (2023)

    2023

  5. [13]

    Advances in Neural Information Processing Systems 36, 28541–28564 (2023)

    Li, C., et al.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36, 28541–28564 (2023)

    2023

  6. [14]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  7. [15]

    In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition

    Lu, M.Y., et al.: Visual language pretrained multiple instance zero-shot transfer for histopathology images. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 19764–19775 (2023)

    2023

  8. [16]

    Nature Medicine30(3), 863–874 (2024) Title Suppressed Due to Excessive Length 17

    Lu, M.Y., et al.: A visual-language foundation model for computational pathology. Nature Medicine30(3), 863–874 (2024) Title Suppressed Due to Excessive Length 17

    2024

  9. [17]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., et al.: Dinov2: Learning robust visual features without super- vision. arXiv preprint arXiv:2304.07193 (2023)

    2023 arXiv

  10. [18]

    Advances in neural information processing systems35, 27730–27744 (2022)

    Ouyang, L., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)

    2022

  11. [19]

    OpenAI blog1(8), 9 (2019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog1(8), 9 (2019)

    2019

  12. [20]

    In: International conference on machine learning

    Radford, A., et al.: Learning transferable visual models from natural lan- guage supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  13. [21]

    Seyfioglu, M.S., et al.: Quilt-llava: Visual instruction tuning by extracting localizednarrativesfromopen-sourcehistopathologyvideos.In:Proceedings oftheIEEE/CVFConferenceonComputerVisionandPatternRecognition. pp. 13183–13192 (2024)

    2024

  14. [22]

    arXiv preprint arXiv:2405.19567 (2024)

    Sun, S., et al.: Dr-llava: Visual instruction tuning with symbolic clinical grounding. arXiv preprint arXiv:2405.19567 (2024)

    2024 arXiv

  15. [23]

    Vaswani,A.,etal.:Attentionisallyouneed.Advancesinneuralinformation processing systems30 (2017)

    2017

  16. [24]

    arXiv preprint arXiv:2309.07778 (2023)

    Vorontsov, E., et al.: Virchow: A million-slide digital pathology foundation model. arXiv preprint arXiv:2309.07778 (2023)

    2023 arXiv

  17. [25]

    In: Pro- ceedings of the 2024 Conference on Human Information Interaction and Retrieval

    Wang, B., Liu, J., Karimnazarov, J., Thompson, N.: Task supportive and personalized human-large language model interaction: A user study. In: Pro- ceedings of the 2024 Conference on Human Information Interaction and Retrieval. pp. 370–375 (2024)

    2024

  18. [26]

    arXiv preprint arXiv:2205.01917 (2022)

    Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)

    2022 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.