Pith. sign in

REVIEW 5 major objections 7 minor 61 references

Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OmniPath, a pathology-specialized vision-language model, claims to outperform every comparison model on all tested diagnostic tasks by steering attention to lesion details and completing slide features with prompt-chosen high-resolution…

desk verdict A plausible systems paper for pathology LVLMs whose headline accuracy claims currently rest on unverified split hygiene and a thin baseline set. read the letter →

arxiv 2412.09521 v3 pith:ZLZN3V7R submitted 2024-12-12 cs.CV cs.AI

Shengxuming Zhang , Weihan Li , Tianhong Gao , Jiacong Hu , Haoming Luo , Xiuming Zhang , Jing Zhang , Mingli Song
show 1 more author
Zunlei Feng
This is my paper · ORCID
classification cs.CVcs.AI
keywords pathologylargevision-languagemodelwholeslideimagefeatureextractionmulti-scalefeaturescancerdiagnosisdetectionandsegmentationvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a single vision-language model can cover the full range of pathology diagnostic work, including cancer subtyping and grading, organ and tissue recognition, invasion and metastasis identification, detection, segmentation, and visual question answering, at accuracies that beat existing pathology LVLMs on essentially every task tested, often surpassing 90%. The proposed fixes target a diagnosed weakness: existing LVLMs concentrate attention on a few key tokens that coarsely summarize the whole image and often sit outside the lesion, losing the local details that diagnosis depends on. OmniPath addresses this with two strategies, mixed task-guided feature enhancement and prompt-guided detail feature completion, trained on roughly 490K samples spanning 21 organs. If the claims hold, a single interactive model could handle tasks that previously required separate specialized systems, while keeping inference fast because only a handful of high-resolution patches are added per slide.

What carries the argument

Mixed task-guided feature enhancement (MGFE) combines hierarchical instruction data at tissue, structure, and cellular levels with three architecture changes: an auxiliary UNI vision encoder pretrained with DINOv2 for fine-grained pathology features, a multi-scale feature fusion that tiles high-resolution inputs and reassembles features so richer detail reaches the LLM without adding image tokens, and a mask encoder and decoder with a new <mask> token so segmentation output is generated directly. Prompt-guided detail feature completion (PGFC) reads the WSI thumbnail, takes the top-S image tokens by attention to locate key regions, extracts high-resolution patches from those regions of the original slide, pools them, and feeds them back into the LLM along with textual position descriptions, completing missing detail without slowing inference. The motivating decision-dependent image token analysis extracts the attention of the final input token over all image tokens and shows that competitors' key tokens aggregate global image semantics while OmniPath's key tokens fall inside annotated lesions.

What would settle it

Request the split policy and rerun the patch-level and slide-level evaluations with a strict patient-level split of the same data; if HCC-S, ICC-G, or LUNG-S accuracies drop dramatically under that split, the reported figures overstate generalization. A complementary check is to run OmniPath on an external WSI cohort from an institution absent from the training list and compare its accuracy with the reported numbers.

Watch

Extended reading notes

Core claim

On its own terms, the paper demonstrates that OmniPath achieves the best performance across all patch-level and slide-level pathological diagnosis tasks evaluated, including cancer subtyping and grading (97.09% on hepatocellular carcinoma subtyping at patch level and 98.40% at slide level), identification of microvascular and neural invasion, pan-cancer classification across 32 types, tumor-infiltrating lymphocyte recognition, and slide-level prognosis prediction. It is also the first model in this comparison that can perform referring detection and segmentation, and it reports detecting cancer cell nuclei inside blood vessels, a reasoning task that requires relating vessels, nuclei, and cancer concepts. The paper attributes the gains to fixing a measured defect: attention analysis shows existing LVLMs channel information through a few image tokens that aggregate global semantics but miss lesion-level detail, whereas in OmniPath the high-attention tokens concentrate inside pathologist-annotated cancer regions.

Load-bearing premise

The evaluations assume that slides or patients were split between training and test sets, but the paper never states this; if patches from the same slides that trained the model were also used to test it, the high accuracies could reflect memorization rather than diagnostic skill.

Editorial extensions

If this is right

  • If the results hold, a single interactive LVLM replaces pipelines of separate classification, detection, and segmentation models for common pathology tasks, with diagnosis presented as dialogue rather than a batch output.
  • The PGFC ablation (average slide-level accuracy 62.81% without it versus 83.91% with it, and 71.17% with randomly chosen patches) implies that where a whole-slide model looks, not just how much it sees, drives slide-level diagnostic accuracy.
  • The MGFE ablation shows that the pathology-pretrained auxiliary encoder contributes more than multi-scale fusion for patch-level diagnosis, while multi-scale fusion matters more for detecting variable-size tissue structures, and both are needed for the full result.
  • Attention analysis suggests the model learns to focus inside pathologist-annotated lesion regions, a property that could make its outputs more auditable than a black-box classifier.
  • The paper's own comparison with specialized vision models reports OmniPath trailing on average for detection (68.05 versus 70.08 F1 against YOLO11) and slightly ahead for segmentation (77.80 versus 76.95 Dice against nnU-Net V2), so the unified model is not yet a wholesale replacement for dedicated detectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The PGFC selection rule is driven by attention over the thumbnail, a mechanism that could be tested on other gigapixel-imagery domains, such as cytology or satellite imaging, where input resolution likewise exceeds model limits.
  • Because the model already emits bounding boxes and masks through dialogue, the same format could be chained into multi-step diagnostic workflows, such as detecting an invasive region and then grading it, without retraining.
  • The gap between OmniPath's zero-shot accuracy (for example, 79.15% on PANDA and 59.33% on CAMELYON17) and its trained accuracy suggests that broadening the training distribution toward external cohorts, not architecture alone, is the next lever for clinical deployment.
  • The reported slide-level accuracies in the 90s rest on the unstated premise that slides or patients were separated between training and test sets, so an independent audit of the split policy is the first check a clinical adopter should perform.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. OmniPath is a pathology-specialized large vision-language model built on LLaVA-1.5, with the vision encoder replaced by SigLIP and augmented by UNI, plus a mask encoder/decoder for segmentation. The paper introduces two strategies: MGFE (mixed task-guided feature enhancement), which trains on detection/segmentation instructions to focus on lesion-level details, and PGFC (prompt-guided detail feature completion), which selects the top-S highest-attention patches from a WSI thumbnail and adds them as high-resolution inputs. The model is trained on a 490K-sample multi-task dataset covering 21 organs. Experiments compare OmniPath against LLaVA-1.5, LLaVA-Med, PA-LLaVA, Quilt-LLaVA (and an author-fine-tuned version) on patch-level and slide-level diagnostics, zero-shot classification, detection, and segmentation, plus ablations of PGFC and MGFE components. The authors report OmniPath as achieving the best accuracy across all in-house diagnostic tasks.

Significance. If the reported results are taken at face value, OmniPath would be a considerable step toward an all-in-one pathology LVLM that handles classification, detection, segmentation, and visual QA at high accuracy, and the PGFC mechanism is an interesting and potentially efficient way to inject WSI-level detail without a computationally heavy patch enumeration. The paper also contributes a large curated instruction-following dataset and an honest limitations section. However, the evaluation as reported is not yet sufficient to establish generalization: the absence of a stated patient/slide-level train-test split, the lack of uncertainty estimates and test-set sizes, and the exclusion of published WSI-capable baselines leave room for alternative explanations of the large accuracy numbers. The central mechanism is plausible and not mathematically circular, but its measured effect needs a more rigorous experimental protocol.

major comments (5)
  1. [Section 5.2 and Appendix F] The manuscript never states the unit of the train/test split (patient, whole slide, patch, or random tile) for any of the internal diagnostic datasets. For patch-level tasks that tile WSIs, if patches from the same slide appear in both training and test sets, the reported accuracies (e.g., HCC-S 97.09% in Table 1) could reflect memorization of slide-specific artifacts rather than learned diagnostic generalization. Appendix F's description of TCGA-Uniform-Tumor 'stratified sampling to extract a subset for training and testing' does not clarify the split unit. Please specify the split policy for every dataset, and provide at least one internal validation with patient- or slide-disjoint splits to support the central claim.
  2. [Tables 1-5 and Appendices D-E] No error bars, confidence intervals, sample sizes, or significance tests are reported for any accuracy, F1-score, or Dice value. Without these, it is impossible to determine whether OmniPath's advantages over Quilt-LLaVA (FT) and other baselines are statistically reliable, particularly on tasks where the gaps are modest (e.g., LUNG-G 71.98 vs. 68.01 in Table 1; DHMC 34.62 vs. 28.48 in Table 3). Report test-set sizes per task and include bootstrap or repeated-run intervals.
  3. [Table 6 ablation] The PGFC ablation exhibits an extraordinary sensitivity to attention-guided patch selection, with HCC-S rising from 20.14% without PGFC to 98.40% with PGFC, and ICC-G from 66.37% to 99.08%. Such a magnitude is difficult to attribute to the mechanism alone unless the selected high-resolution patches contain the true discriminative signal; without a verified patient/slide-level split, this result is also consistent with test-set contamination of the selected patches. The random-selection control (HCC-S 79.46%) helps, but it does not rule out leakage through attention-based selection that correlates with the test label. Please report the overlap statistics between selected patches and training patches and rerun this ablation on slide-disjoint splits.
  4. [Section 2 and Section 5.2] The manuscript discusses PathAlign, PRISM, PathAsst, and PathMMU as existing pathology LVLMs but never evaluates them. The claim in Section 5.2 that 'OmniPath achieves the best performance across all patch-level and slide-level pathological diagnosis tasks' is therefore only validated against LLaVA-derived baselines, one of which (Quilt-LLaVA) was fine-tuned by the authors, plus a small subset of closed-source models in Appendix E. Adding at least one established WSI-capable system (e.g., PathAlign or PRISM) to the comparisons, even on a limited set of tasks, would materially strengthen the claim.
  5. [Section 4.2] The PGFC algorithm states that 'we remove elements corresponding to image background regions from Ψ(N+M),ev' before selecting top-S patches, but no threshold or criterion for defining background is given. This is a free parameter that affects which patches are selected and thus the reported results; specify the detection rule (e.g., attention percentile, Otsu, or a fixed threshold) and the exact values used.
minor comments (7)
  1. [Section 5.1] The training description omits several standard hyperparameters (warmup steps, weight decay, per-GPU batch size, and total trainable parameter count); adding these would improve reproducibility.
  2. [Section 3.2 and Figure 2] The text says attention values are extracted from 'the input layer of M' while the figure caption says 'the first layer of the LLM'; this discrepancy should be reconciled.
  3. [Table 3 and reference [46]] The NCT-CRC dataset is referenced to Macenko et al., which is a color-normalization paper rather than the source of the NCT-CRC-HE-100K dataset; the citation should be corrected to the proper Kather et al. reference.
  4. [Appendix F] Table 13 reports 108,806 samples with 'unsure' organ origin, over one fifth of the dataset; clarify how these samples are used and whether they affect organ-classification results.
  5. [Appendix A] There is a typo: 'In the The heatmaps generated by LLaVA-Med...' should read 'The heatmaps generated by...'.
  6. [Appendix E] The closed-source comparison reports only a subset of tasks; state the number of images per task and the API parameters (e.g., temperature, max tokens) used for evaluation.
  7. [Section 5.3] The claim that CCRCC, MHIST, NCT-CRC, PANDA, DHMC, and CAMELYON17 were 'not included in the training set' is plausible, but Appendix F shows that TCGA-CRC-DX (used for MSI training) and NCT-CRC are both colorectal-cancer datasets; please confirm no patch-level overlap between training and zero-shot sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: OmniPath's reported performance is empirical benchmarking and ablation, not a prediction forced by construction or by load-bearing self-citation.

full rationale

The paper's central claim is that OmniPath 'achieves the best performance across all patch-level and slide-level pathological diagnosis tasks' (Section 5.2). This is an empirical comparison against other models on held-out test sets; the numbers are measurements, not quantities derived from the method's own definitions. The two proposed strategies, MGFE and PGFC, are architectural and data-level interventions whose effects are evaluated by direct comparison and ablation. PGFC does use the model's own attention to select top-S high-resolution patches, but this is a self-referential mechanism, not circular reasoning: the selected patches are real image content, the selection does not use test labels, and the ablation includes a random-patch control to separate the effect of attention-guided selection from mere added resolution. The paper's self-citations (references [1]-[3] for prior nuclei, vessel, and microvascular invasion models) appear only as background in Related Work and Section 4.1; they are not used to justify the central performance claims or to exclude alternative methods. The lack of an explicit patient- or slide-level train/test split for some self-collected data is a legitimate generalization-risk concern, but it is an evaluation validity issue, not a circularity in the derivation chain. No equation is shown to reduce a predicted quantity to a fitted parameter or to an input by construction. Therefore, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the empirical setup rather than a mathematical derivation. The main free parameters are the attention-based zooming hyperparameters (S, resolution schedule, background criterion). The key axioms are that attention reflects diagnostic relevance and that the training data are representative and properly split. No new physical or symbolic entities are introduced.

free parameters (3)
  • PGFC patch count S = 8
    Number of high-resolution patches selected by attention in PGFC (Sec 4.2, Sec 5.1); set by hand without sensitivity analysis.
  • multi-scale input resolution schedule = not specified
    The multi-scale feature fusion uses a series of integer-multiple resolutions of the base vision encoder input (Sec 4.1); the exact set is not listed, and the choice is not justified by an experiment.
  • attention background removal threshold = not specified
    PGFC removes 'image background regions' from the attention vector before top-S selection (Sec 4.2); the criterion for background is not defined.
assumptions (4)
  • domain assumption The attention weights of the LLM's input layer are a faithful proxy for the visual evidence used in the model's decision.
    Section 3.2 builds the motivation on attention heatmaps, assuming high attention implies diagnostic relevance; this is an interpretive assumption, not a proven property.
  • domain assumption The 490K training samples are representative of the test distribution and are split at the patient or WSI level so that no patches from a test slide appear in training.
    The accuracy claims depend on this unstated split policy; Section 5.2 and Appendix F do not describe it.
  • domain assumption Fusing SigLIP (global) and UNI (fine-grained) features by average-pooling and channel concatenation preserves complementary information without degrading the LLM's language understanding.
    Section 4.1 describes the fusion; only ablations support it, with no analysis of potential interference.
  • domain assumption The pretrained LLaVA-1.5 backbone, SigLIP, and UNI encoders provide a valid foundation, and their pretraining distributions overlap with the pathology data.
    These pretrained models are used as-is; the paper does not validate their suitability beyond the reported fine-tuning results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis." pith.science (2026). https://pith.science/paper/ZLZN3V7R

@misc{pith2026241209521,
  author       = {Pith},
  title        = {Pith review of: Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLZN3V7R}},
  note         = {Machine review of arXiv:2412.09521}
}
read the original abstract

Pathological diagnosis is vital for determining disease characteristics, guiding treatment, and assessing prognosis, relying heavily on detailed, multi-scale analysis of high-resolution whole slide images (WSI). However, existing large vision-language models (LVLMs) are limited by input resolution constraints, hindering their efficiency and accuracy in pathology image analysis. To overcome these issues, we propose two innovative strategies: the mixed task-guided feature enhancement, which directs feature extraction toward lesion-related details across scales, and the prompt-guided detail feature completion, which integrates coarse- and fine-grained features from WSI based on specific prompts without compromising inference speed. Leveraging a comprehensive dataset of 490K samples from diverse pathology tasks, we trained the pathology-specialized LVLM, OmniPath. Extensive experiments demonstrate that this model significantly outperforms existing methods in diagnostic accuracy and efficiency, providing an interactive, clinically aligned approach for auxiliary diagnosis in a wide range of pathology applications.

Figures

Figures reproduced from arXiv: 2412.09521 by the authors.

Figure 1
Figure 1. Dialogue examples of our OmniPath, a vision-language model optimized for pathology, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The green contours on the pathology slides mark cancerous regions annotated by pathol￾ogists. The first column shows the attention distri￾bution heatmap of the LLM’s final input token over all image tokens, where the intensity of attention values is mapped from blue (low) to red (high). In each row showing different model results, a red box and a yellow box are used to select a key to￾ken (with relatively high atten… view at source ↗
Figure 3
Figure 3. Overview of the proposed OmniPath. Left: the architecture of OmniPath with the MGFE [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: More samples of the attention of upcoming token on image tokens (like the fist column in [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: t-SNE visualization results of slide-level image features extracted by vision encoders of [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 46 canonical work pages

  1. [1]

    Edge-competing pathological liver vessel segmentation with limited labels,

    Z. Feng, Z. Wang, X. Wang,et al., “Edge-competing pathological liver vessel segmentation with limited labels,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 1325–1333

    work page 2021

  2. [2]

    Mutual-complementing framework for nuclei detection and segmentation in pathology image,

    Z. Feng, Z. Wang, X. Wang,et al., “Mutual-complementing framework for nuclei detection and segmentation in pathology image,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4036–4045

    work page 2021

  3. [3]

    A loopback network for explainable microvascular invasion classification,

    S. Zhang, T. Shi, Y . Jiang,et al., “A loopback network for explainable microvascular invasion classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2023, pp. 7443–7453

    work page 2023

  4. [4]

    Scaling vision transformers to gigapixel images via hierar- chical self-supervised learning,

    R. J. Chen, C. Chen, Y . Li,et al., “Scaling vision transformers to gigapixel images via hierar- chical self-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 144–16 155

    work page 2022

  5. [5]

    Towards a general-purpose foundation model for computational pathology,

    R. J. Chen, T. Ding, M. Y . Lu, et al., “Towards a general-purpose foundation model for computational pathology,” Nature Medicine, vol. 30, no. 3, pp. 850–862, 2024

    work page 2024

  6. [6]

    A whole-slide foundation model for digital pathology from real-world data,

    H. Xu, N. Usuyama, J. Bagga, et al., “A whole-slide foundation model for digital pathology from real-world data,” Nature, pp. 1–8, 2024

    work page 2024

  7. [7]

    A foundation model for clinical-grade computa- tional pathology and rare cancers detection,

    E. V orontsov, A. Bozkurt, A. Casson,et al., “A foundation model for clinical-grade computa- tional pathology and rare cancers detection,” Nature medicine, pp. 1–12, 2024

    work page 2024

  8. [8]

    A pathology foundation model for cancer diagnosis and prognosis prediction,

    X. Wang, J. Zhao, E. Marostica, et al., “A pathology foundation model for cancer diagnosis and prognosis prediction,” Nature, pp. 1–9, 2024

    work page 2024

Show all 61 references
  1. [9]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  2. [10]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, et al. , “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024, pp. 26 296–26 306

    2024

  6. [14]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  7. [15]

    Shikra: Unleashing multimodal llm’s referential dialogue magic,

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,”arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  8. [16]

    Lisa: Reasoning segmentation via large language model,

    X. Lai, Z. Tian, Y . Chen,et al., “Lisa: Reasoning segmentation via large language model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9579–9589

    2024

  9. [18]

    Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology,

    Y . Sun, H. Wu, C. Zhu,et al., “Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology,” in European Conference on Computer Vision, Springer, 2025, pp. 56–73

    2025

  10. [20]

    A multimodal generative ai copilot for human pathology,

    M. Y . Lu, B. Chen, D. F. Williamson,et al., “A multimodal generative ai copilot for human pathology,” Nature, pp. 1–3, 2024

    2024

  11. [21]

    Pathalign: A vision-language model for whole slide images in histopathology,

    F. Ahmed, A. Sellergren, L. Yang,et al., “Pathalign: A vision-language model for whole slide images in histopathology,” arXiv preprint arXiv:2406.19578, 2024

    2024 arXiv

  12. [22]

    Pa-llava: A large language-vision assistant for human pathology image understanding,

    D. Dai, Y . Zhang, L. Xu, et al. , “Pa-llava: A large language-vision assistant for human pathology image understanding,” arXiv preprint arXiv:2408.09530, 2024. 10

    2024 arXiv

  13. [23]

    Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration,

    Y . Sun, Y . Zhang, Y . Si,et al. , “Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration,” arXiv preprint arXiv:2407.00203, 2024

    2024 arXiv

  14. [24]

    Prism: A multi-modal generative foundation model for slide-level histopathology,

    G. Shaikovski, A. Casson, K. Severson, et al., “Prism: A multi-modal generative foundation model for slide-level histopathology,” arXiv preprint arXiv:2405.10254, 2024

    2024 arXiv

  15. [25]

    Multiple instance captioning: Learning representations from histopathology textbooks and articles,

    J. Gamper and N. Rajpoot, “Multiple instance captioning: Learning representations from histopathology textbooks and articles,” in Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2021, pp. 16 549–16 559

    2021

  16. [27]

    Wsi-vqa: Interpreting whole slide images by generative visual question answering,

    P. Chen, C. Zhu, S. Zheng, H. Li, and L. Yang, “Wsi-vqa: Interpreting whole slide images by generative visual question answering,” in European Conference on Computer Vision, Springer, 2025, pp. 401–417

    2025

  17. [28]

    A visual–language foun- dation model for pathology image analysis using medical twitter,

    Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, and J. Zou, “A visual–language foun- dation model for pathology image analysis using medical twitter,”Nature medicine, vol. 29, no. 9, pp. 2307–2316, 2023

    2023

  18. [29]

    Quilt-1m: One million image-text pairs for histopathology,

    W. Ikezogwo, S. Seyfioglu, F. Ghezloo, et al., “Quilt-1m: One million image-text pairs for histopathology,” Advances in neural information processing systems, vol. 36, 2024

    2024

  19. [30]

    Attention-based deep multiple instance learning,

    M. Ilse, J. Tomczak, and M. Welling, “Attention-based deep multiple instance learning,” in International conference on machine learning, PMLR, 2018, pp. 2127–2136

    2018

  20. [31]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification,

    Z. Shao, H. Bian, Y . Chen, Y . Wang, J. Zhang, X. Ji,et al., “Transmil: Transformer based correlated multiple instance learning for whole slide image classification,” Advances in neural information processing systems, vol. 34, pp. 2136–2147, 2021

    2021

  21. [32]

    Camel: A weakly supervised learning framework for histopathol- ogy image segmentation,

    G. Xu, Z. Song, Z. Sun,et al., “Camel: A weakly supervised learning framework for histopathol- ogy image segmentation,” in Proceedings of the IEEE/CVF International Conference on computer vision, 2019, pp. 10 682–10 691

    2019

  22. [33]

    Differentiable zooming for multiple instance learning on whole-slide images,

    K. Thandiackal, B. Chen, P. Pati, et al., “Differentiable zooming for multiple instance learning on whole-slide images,” inEuropean Conference on Computer Vision, Springer, 2022, pp. 699– 715

    2022

  23. [34]

    Pathinsight: Instruction tuning of multimodal datasets and models for intelligence assisted diagnosis in histopathology,

    X. Wu, R. Xu, P. Wei,et al., “Pathinsight: Instruction tuning of multimodal datasets and models for intelligence assisted diagnosis in histopathology,” arXiv preprint arXiv:2408.07037, 2024

    2024 arXiv

  24. [35]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, 2024

    2024

  25. [36]

    Learning transferable visual models from natu- ral language supervision,

    A. Radford, J. W. Kim, C. Hallacy, et al., “Learning transferable visual models from natu- ral language supervision,” in International conference on machine learning, PMLR, 2021, pp. 8748–8763

    2021

  26. [37]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730– 27 744, 2022

    2022

  27. [38]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning, PMLR, 2023, pp. 19 730–19 742

    2023

  28. [40]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni,et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  29. [41]

    Sigmoid loss for language image pre- training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre- training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 975–11 986

    2023

  30. [42]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770– 778

    2016

  31. [43]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026. 11

    2023

  32. [44]

    Integrative analysis of histological textures and lymphocyte infiltration in renal cell carcinoma using deep learning,

    O. Brummer, P. Pölönen, S. Mustjoki, and O. Brück, “Integrative analysis of histological textures and lymphocyte infiltration in renal cell carcinoma using deep learning,” bioRxiv, pp. 2022–08, 2022

    2022

  33. [45]

    A petri dish for histopathology image analysis,

    J. Wei, A. Suriawinata, B. Ren, et al., “A petri dish for histopathology image analysis,” in Artificial Intelligence in Medicine: 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, June 15–18, 2021, Proceedings , Springer, 2021, pp. 11–24

    2021

  34. [46]

    A method for normalizing histology slides for quantitative analysis,

    M. Macenko, M. Niethammer, J. S. Marron, et al., “A method for normalizing histology slides for quantitative analysis,” in 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009, pp. 1107–1110. DOI: 10.1109/ISBI.2009.5193250

    2009

  35. [47]

    Artificial intelligence for diagnosis and gleason grading of prostate cancer: The panda challenge,

    W. Bulten, K. Kartasalo, P.-H. C. Chen, et al., “Artificial intelligence for diagnosis and gleason grading of prostate cancer: The panda challenge,”Nature medicine, vol. 28, no. 1, pp. 154–163, 2022

    2022

  36. [48]

    Pathologist- level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks,

    J. W. Wei, L. J. Tafe, Y . A. Linnik, L. J. Vaickus, N. Tomita, and S. Hassanpour, “Pathologist- level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks,” Scientific reports, vol. 9, no. 1, p. 3358, 2019

    2019

  37. [49]

    1399 h&e-stained sentinel lymph node sections of breast cancer patients: The camelyon dataset,

    G. Litjens, P. Bandi, B. Ehteshami Bejnordi,et al., “1399 h&e-stained sentinel lymph node sections of breast cancer patients: The camelyon dataset,” GigaScience, vol. 7, no. 6, giy065, 2018

    2018

  38. [53]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, et al., “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” in Advances in Neural Information Processing Systems, vol. 36, Curran Associates, Inc., 2023, pp. 28 541–28 564

    2023

  39. [54]

    Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos,

    M. S. Seyfioglu, W. O. Ikezogwo, F. Ghezloo, R. Krishna, and L. Shapiro, “Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20...

    2024

  40. [55]

    Jocher and J

    G. Jocher and J. Qiu, Ultralytics yolo11, version 11.0.0, 2024. [Online]. Available: https: //github.com/ultralytics/ultralytics

    2024

  41. [56]

    Nnu-net: A self- configuring method for deep learning-based biomedical image segmentation,

    F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “Nnu-net: A self- configuring method for deep learning-based biomedical image segmentation,” Nature methods, vol. 18, no. 2, pp. 203–211, 2021

    2021

  42. [57]

    A multi-organ nucleus segmentation challenge,

    N. Kumar, R. Verma, D. Anand, et al., “A multi-organ nucleus segmentation challenge,” IEEE transactions on medical imaging, vol. 39, no. 5, pp. 1380–1391, 2019

    2019

  43. [58]

    Conic challenge: Pushing the frontiers of nu- clear detection, segmentation, classification and counting,

    S. Graham, Q. D. Vu, M. Jahanifar, et al., “Conic challenge: Pushing the frontiers of nu- clear detection, segmentation, classification and counting,” Medical image analysis, vol. 92, p. 103 047, 2024

    2024

  44. [59]

    N. P. Jack, W. Thomas, L. Marick, and R. Fabien,Segmentation of Nuclei in Histopathology Images by deep regression of the distance map, version 1.1, Zenodo, Feb. 2019.DOI: 10.5281/ zenodo.2579118. [Online]. Available: https://doi.org/10.5281/zenodo.2579118

    2019 doi

  45. [60]

    Nucls: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer,

    M. Amgad, L. A. Atteya, H. Hussein, et al., “Nucls: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer,” GigaScience, vol. 11, giac037, 2022

    2022

  46. [61]

    Pannuke dataset extension, insights and baselines,

    J. Gamper, N. A. Koohbanani, K. Benes, et al., “Pannuke dataset extension, insights and baselines,” arXiv preprint arXiv:2003.10778, 2020

    2003 arXiv

  47. [62]

    Nuinsseg: A fully annotated dataset for nuclei instance segmentation in h&e-stained histological images,

    A. Mahbod, C. Polak, K. Feldmann, et al., “Nuinsseg: A fully annotated dataset for nuclei instance segmentation in h&e-stained histological images,” Scientific Data, vol. 11, no. 1, p. 295, 2024

    2024

  48. [63]

    Artificial intelligence for tumour tissue detection and histological regression grading in oesophageal adenocarcinomas: A retrospective algorithm development and validation study,

    Y . Tolkach, L. M. Wolgast, A. Damanakis,et al., “Artificial intelligence for tumour tissue detection and histological regression grading in oesophageal adenocarcinomas: A retrospective algorithm development and validation study,”The Lancet Digital Health, vol. 5, no. 5, e265–...

    2023

  49. [64]

    Deep learning-based mapping of tumor infiltrating lymphocytes in whole slide images of 23 types of cancer,

    S. Abousamra, R. Gupta, L. Hou, et al., “Deep learning-based mapping of tumor infiltrating lymphocytes in whole slide images of 23 types of cancer,” Frontiers in oncology, vol. 11, p. 806 603, 2022

    2022

  50. [65]

    J. N. Kather, Histological image tiles for TCGA-CRC-DX, color- normalized, sorted by MSI status, train/test split , Zenodo, May 2020. DOI: 10 . 5281 / zenodo . 3832231. [Online]. Available: https://doi.org/10.5281/zenodo.3832231

    2020 doi

  51. [66]

    Universal encoding of pan-cancer histology by deep texture representations,

    D. Komura, A. Kawabe, K. Fukuta, et al., “Universal encoding of pan-cancer histology by deep texture representations,” Cell Reports, vol. 38, no. 9, 2022

    2022

  52. [67]

    Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology,

    Y . Sun, C. Zhu, S. Zheng, et al., “Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 5034–5042

    2024

  53. [68]

    Pathvqa: 30000+ questions for medical visual question answering,

    X. He, Y . Zhang, L. Mou, E. Xing, and P. Xie, “Pathvqa: 30000+ questions for medical visual question answering,” arXiv preprint arXiv:2003.10286, 2020. 26

    2003 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.