Pith. sign in

REVIEW 3 major objections 4 minor 26 references

Explaining Digital Pathology Models via Clustering Activations

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Clustering the final convolutional layer's activations with non-negative matrix factorization produces interpretable whole-slide segmentations that reveal the morphological features a prostate cancer model uses.

desk verdict A clean, modest NMF-clustering explanation tool for a prostate-CNN, undermined by circular quantitative validation; the pathologist's qualitative reading is the real evidence. read the letter →

arxiv 2511.14558 v2 pith:EXG7S7WC submitted 2025-11-18 cs.CV

Adam Bajger , Jan Obdržálek , Vojtěch Kůr , Rudolf Nenutil , Petr Holub , Vít Musil , Tomáš Brázdil This is my paper · ORCID
classification cs.CV
keywords explainableAIdigitalpathologynon-negativematrixfactorizationactivationclusteringwholeslideimagesprostatecancersemanticsegmentationCNNinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to explain a CNN-based digital pathology model by clustering the activation vectors of its last convolutional layer. Using non-negative matrix factorization, the method turns those activations into a set of interpretable classes that segment the whole slide into morphological regions. On a prostate cancer model, pathologist inspection found the classes correspond to structures such as dense nuclear chains, small lumina, and tissue edges, and a linear regression showed that the cancer-assigned classes are the same ones that drive the model's predictions. The technique offers a global, fine-grained alternative to saliency maps, potentially making AI pathology models easier to trust and adopt.

What carries the argument

The central mechanism is non-negative matrix factorization of feature vectors extracted at each spatial location of the final convolutional layer. NMF approximates the matrix of feature vectors V as W H, where the K rows of H act as class prototype vectors and W stores per-location class intensities; each pixel is assigned the class with maximal weight. To handle whole-slide images, overlapping tile contributions are averaged per location and tissue-free tiles are discarded. The factorization is trained on a subset of slides, then applied to fresh slides with H fixed.

What would settle it

Take the same prostate slides and replace the model's predictions with independent pathologist annotations of cancer regions as the target for the logistic regression. If the coefficients for the cancer-associated clusters (dense nuclei chains and small lumina) do not stay positive and significant, the claim that these clusters capture the morphology of cancer is falsified. Equivalently, train the clustering on a model with deliberately corrupted or shuffled final-layer features; the semantic alignment should disappear.

Watch

Extended reading notes

Core claim

The paper claims that clustering activation vectors from the deepest convolutional layer of a whole-slide CNN with NMF produces a semantic segmentation that reflects the model's decision-relevant morphology, not just its hotspot regions. On a prostate cancer classifier, six clusters separated into cancer-associated structures (chains of densely packed nuclei and small lumina) and benign structures (fascicular tissue and edges). Quantitative checks show that a logistic regression using only per-class activation weights reconstructs the model's cancer calls with high accuracy (0.984), and the same cancer-positive classes correlate strongly with GradCAM heatmaps while providing finer spatial de

Load-bearing premise

The quantitative evaluation uses the model's own cancer predictions as ground truth, so the logistic regression measures how well clusters explain the model, not how well they match independent pathological truth.

Editorial extensions

If this is right

  • Pathologists can inspect global model behaviour across an entire slide, not just single predictions.
  • The clusters refine information in saliency maps such as GradCAM, locating distinct morphological patterns within highlighted regions.
  • The method needs only a trained classifier and a set of unlabeled tiles, so it can be applied to existing clinical models.
  • The produced segmentations can serve as a sanity check that the model relies on known cancer morphology.
  • Trust in CNN-based pathology tools may increase, supporting clinical adoption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the method operates on any spatial feature map, it could be applied to transformer-based foundation models in pathology, provided their features are made non-negative or are transformed appropriately.
  • The cluster structure could be used as an audit tool to detect spurious correlations: if a cluster corresponds to an artifact such as stain or tissue fold rather than a biological structure, the model may be overfitted to that artifact.
  • A direct extension would be to evaluate on a model known to be wrong on certain slides; the clusters should still track the model's internal features rather than true pathology, clarifying that the method explains the model, not the disease.
  • The K parameter acts as a granularity dial; future work could relate optimal K to the number of morphological tissue types present.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a post-hoc explainability method for CNN-based digital pathology models. Feature vectors from the last convolutional layer of a trained classifier are clustered with non-negative matrix factorization (NMF); the resulting cluster weights are visualized as heatmaps and a hard clustering overlay, providing a global, semantically segmented view of what the model "sees." The method is demonstrated on a VGG-16 prostate cancer WSI classifier [9]. A resident pathologist qualitatively described the six clusters as corresponding to morphological structures (e.g., dense nuclear chains, small lumina, tissue edges). Quantitative support is attempted via a logistic regression that predicts the model's own cancer call from cluster weights, plus comparisons with GradCAM. The paper claims the method yields global explanations that could increase clinical trust and adoption.

Significance. If the central claim is established, the method would be a useful complement to saliency maps: it produces global, multi-class, whole-slide visualizations from a classifier not trained to segment, with potential value for model auditing and for communicating model behavior to pathologists. The NMF-on-activations approach is simple and plausibly generalizes to other architectures. However, the current manuscript does not establish the load-bearing claim that the clusters correspond to human-meaningful morphology: the only direct evidence is a single pathologist's qualitative reading, and the quantitative evaluation is self-referential. The idea is promising, but the evaluation must be strengthened before the claim can be accepted.

major comments (3)
  1. [§3.2, Table 1] The quantitative validation is circular. The logistic regression is trained "using the models' predictions (1 indicates cancer) as the ground truth." The model's prediction is a deterministic function of the last-convolutional-layer features after global max-pooling and an FC layer; the NMF clusters are computed on exactly those same post-ReLU features (§2.1). A linear model on NMF cluster weights can therefore reconstruct the classifier's decision almost perfectly even if the clusters have no morphological meaning. The AUC of 0.998 is expected self-consistency, not evidence that clusters 2, 4, 5 correspond to carcinomatous structures. Independent ground truth is available: §2.3 states that all positive biopsies were manually annotated with cancer polygons. The logistic regression (or a similar analysis) should be evaluated against those annotations, not against the model's own output.
  2. [§3.1] The only evidence tying clusters to morphology is the resident pathologist's qualitative description of the K=6 heatmaps. No inter-rater agreement is reported, no quantitative mapping to the manual cancer annotations is provided, and no assessment of intra-observer reliability is given. The statement "This suggests that the classes correspond to observable morphological structures" is plausible but anecdotal. The claim is load-bearing for the paper's central contribution; it requires either multiple independent pathologists with measured agreement or a quantitative comparison with the annotated cancer regions.
  3. [§3.2, Fig. 4] The correlation and cosine-similarity analyses in Fig. 4 measure internal consistency among cluster weights/vectors. They confirm that clusters 2 and 5 (and 4) are mutually similar and distinct from clusters 1,3,6, but they do not establish what those clusters semantically represent. The subsequent conclusion that "classes 2, 4, and 5 highlight pro-cancer features" is derived from the circular logistic regression and is therefore not supported by this analysis. Please provide an external semantic target for the clusters, such as manual cancer annotations or pathologist-labeled regions per cluster.
minor comments (4)
  1. [§2.3] The sentence "yielding the positive/negative ratio of WSIs of 37/50" is ambiguous: are 37 positive and 50 negative? The text later says the test dataset has 87 WSIs, which matches 37+50, but the phrasing should be clarified.
  2. [§2.1] Minor grammar: "On training time" should be "At training time" or "During training."
  3. [§3.2] Table 1 reports point estimates for accuracy, precision, recall, F1, and AUC without confidence intervals or a description of how these were computed (per-tile? per-slide? across test folds?). Since the paper emphasizes quantitative verification, error bars or a resampling procedure would strengthen the presentation.
  4. [§3.3, Table 2] The low IoU between GradCAM-positive regions and positive-weight classes is interpreted as "more nuanced information." An alternative interpretation is that GradCAM is not faithful for this model. The paper should discuss this possibility or provide evidence that the clusters refine, rather than contradict, the saliency signal.

Circularity Check

1 steps flagged · score 5.0 of 10

Quantitative validation in §3.2 is self-referential: the logistic regression predicts the same model's label from cluster weights computed on that model's own activations.

  1. fitted input called prediction [Section 3.2, Table 1]
    "For the second test, we trained a logistic regression model that approximates the model’s behavior based only on the information about the clustering. We used the models’ predictions (1 indicates cancer) as the ground truth and trained the linear model to predict cancer positivity from the sum of all weights of each class. ... We confirm that classes 2, 4, and 5 highlight pro-cancer features."

    The target variable is the output of the very model being explained, and the NMF cluster weights are computed from that same model's last-convolutional activations. NMF reconstructs V as WH, so W is a low-rank linear encoding of the feature vectors that feed the classifier. A logistic regression on W can therefore recover the model's decision without any independent morphological ground truth; the 0.998 AUC demonstrates self-consistency between two views of the same features, not correspondence to cancer morphology. Calling these classes 'pro-cancer' is essentially relabeling the model's own predictions through the fitted regression, rather than validating against external cancer annotations (which the dataset already contains).

full rationale

The core clustering procedure itself is not circular: NMF factorization of activation vectors is an unsupervised, self-contained operation, and the qualitative comparison with GradCAM and the pathologist's descriptions provide some independent content. The circularity is concentrated in §3.2, where the quantitative 'cancer sensitivity' test uses the model's own predictions as ground truth while the cluster weights are derived from the same activations that determine those predictions. This makes the reported AUC/accuracy a consistency check rather than an external validation of morphological correspondence. The paper's central claim is not entirely reduced to a tautology because it also relies on the pathologist's qualitative assessment and the cited slide-level accuracy of the prior model [9], but that prior model shares an author (Nenutil) and is used as the object of explanation, not as independent evidence for the clustering method. No load-bearing uniqueness or ansatz is imported from self-citations. Given that the main quantitative confirmation is partially circular, a score of 5 is appropriate.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The method introduces no physical entities. It does introduce learned cluster definitions (H) and semantic labels; these are fitted to the model's activations and validated only against the model's own outputs plus one pathologist's reading.

free parameters (6)
  • Number of clusters K = 4, 6, 8
    Chosen by hand; higher K gives finer clusters (§2.3). Results focus on K=6; no principled selection criterion.
  • Tile outline margin = 2 px
    Feature vectors near tile border are discarded by omitting a 2-px outline to reduce tiling artifacts (§2.2); chosen by hand.
  • Tissue coverage threshold = not specified
    'Sufficient amount of tissue' is not quantified, so slide prefiltering is under-specified (§2.2).
  • NMF class vectors H and weights W = fitted on 37,332 tiles
    Learned by minimizing the Frobenius norm (§2.1); central output of the method, fitted to training activations.
  • Logistic regression coefficients = Table 1 values
    Trained to predict the model's own cancer labels from cluster weight sums (§3.2); coefficients drive the claim that classes 2, 4, 5 are cancer-associated.
  • Training slide selection = 16 positive + 8 negative slides
    Manual selection of clustering training slides (§2.3); may affect which clusters emerge.
assumptions (5)
  • domain assumption Convolutional networks are mostly equivariant, so feature vectors from overlapping tiles of the same patch should be similar
    Invoked in §2.2 to justify averaging feature vectors across tiles; the paper itself notes this is 'not guaranteed, especially near the tile border'.
  • standard math ReLU-applied last-layer activations are non-negative and adequate input for NMF
    §2.1 uses ReLU outputs to satisfy NMF non-negativity; standard practice.
  • domain assumption The prostate model of [9] is accurate enough that its predictions can serve as ground truth for cancer
    §3.2 explicitly uses model predictions as ground truth for quantitative validation; relies on 100% slide-level accuracy reported in [9].
  • domain assumption Euclidean NMF reconstruction of feature vectors produces semantically meaningful clusters
    Core method assumption; supported only by one pathologist's qualitative inspection (§3.1).
  • domain assumption Final convolutional layer features encode morphology relevant to classification
    The method clusters the deepest conv layer (§2.3) and assumes those features carry tissue-structure information.
invented entities (1)
  • NMF semantic classes (e.g., 'dense nuclear chains', 'small lumina')
    purpose: Give morphological labels to activation clusters for model explanation
    Classes are derived from the model's own activations; their morphological meaning is assigned by a single pathologist's inspection, with no external benchmark or inter-rater study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explaining Digital Pathology Models via Clustering Activations." pith.science (2026). https://pith.science/paper/EXG7S7WC

@misc{pith2026251114558,
  author       = {Pith},
  title        = {Pith review of: Explaining Digital Pathology Models via Clustering Activations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXG7S7WC}},
  note         = {Machine review of arXiv:2511.14558}
}
read the original abstract

We present a clustering-based explainability technique for digital pathology models based on convolutional neural networks. Unlike commonly used methods based on saliency maps, such as occlusion, GradCAM, or relevance propagation, which highlight regions that contribute the most to the prediction for a single slide, our method shows the global behaviour of the model under consideration, while also providing more fine-grained information. The result clusters can be visualised not only to understand the model, but also to increase confidence in its operation, leading to faster adoption in clinical practice. We also evaluate the performance of our technique on an existing model for detecting prostate cancer, demonstrating its usefulness.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 3 linked inside Pith

  1. [13]

    Glea- son grading: past, present and future,

    B. Delahunt, R. J. Miller, J. R. Srigleyet al., “Glea- son grading: past, present and future,”Histopathology, vol. 60, no. 1, pp. 75–86, Jan. 2012

    2012

  2. [14]

    Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,

    G. Campanella, M. G. Hanna, L. Geneslawet al., “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,”Na- ture Medicine, vol. 25, no. 8, pp. 1301–1309, 2019

    2019

  3. [9]

    This suggests that the classes correspond to observable mor- phological structures

    Tissue edges plus limited surroundings. This suggests that the classes correspond to observable mor- phological structures. Fig. 2: Each of the 6 classes from Figure 1 visualised sepa- rately. Order is left-to-right. Classes are colored 1■, 2■, 3■, 4■, 5■, and 6■. 3.2. Quantitative analysis We performed two tests to quantitatively verify these findings. O...

  4. [1]

    look similar

    INTRODUCTION The prevalence of oncological diseases is increasing all around the world, and early detection of cancer is crucial for improving treatment outcomes and survival rates. Digital pathology, the contemporary approach to cancer detection, involves analysing digitized copies of tissue samples, called Whole Slide Images (WSIs). WSIs are then displa...

    arXiv 2025

  5. [2]

    MATERIALS AND METHODS Our method is intended for digital pathology models based on convolutional neural networks (CNNs). These networks take as input a tile from a histopathology slide and produce a prediction, which is typically one of predefined classes (e.g., healthy/cancerous) or a numerical value. 2.1. Cluster computation Let us fix a CNN model, a co...

  6. [3]

    1 shows a typical output of our model forK= 6, where each of the classes is shown in a different color and classes are overlaid over the top of each other

    RESULTS AND DISCUSSION Fig. 1 shows a typical output of our model forK= 6, where each of the classes is shown in a different color and classes are overlaid over the top of each other. Fig. 3b shows the intensity heatmap of all classes, and Fig. 3c shows clustering on the same slide. Fig. 2 shows the class intensities individually. The pathologist observed...

  7. [4]

    Fascicular structures with sparse elongated nuclei

  8. [5]

    Chains of densely packed nuclei; in tumour-bearing slides this pattern corresponds to carcinomatous areas, while in non-tumorous slides, it also captures normal epithelium

Show all 26 references
  1. [6]

    Similar to and overlapping with class 1, with emphasis on sparse tissue structure

  2. [7]

    Circular and/or semicircular small holes plus limited sur- roundings; this class is highly sensitive to very small lu- mina (carcinoma)

  3. [8]

    Similar to and overlapping with class 2

  4. [10]

    Unlike commonly used techniques such as Grad- CAM, the provided explanations focus on the global behavior of the model

    CONCLUSION Our clustering-based explainability technique offers new in- sights into the inner workings of CNN-based digital pathol- ogy models. Unlike commonly used techniques such as Grad- CAM, the provided explanations focus on the global behavior of the model. The explanati...

  5. [11]

    MOU 385 920

    COMPLIANCE WITH ETHICAL STANDARDS The project was approved by the Ethical Committee of Masaryk Memorial Cancer Institute, No. MOU 385 920

  6. [12]

    101079183 (BioMedAI TWINNING)

    ACKNOWLEDGMENTS The work utilized the infrastructure and AI framework de- veloped BioMedAI, supported by the European Union’s Horizon Europe research and innovation program under No. 101079183 (BioMedAI TWINNING). Computational re- sources were provided by the e-INFRA CZ proje...

  7. [15]

    A survey on deep learning in medical image analysis,

    G. Litjens, T. Kooi, B. E. Bejnordiet al., “A survey on deep learning in medical image analysis,”Medical Im- age Analysis, vol. 42, pp. 60–88, 2017

    2017

  8. [16]

    Explainable artificial intelligence (XAI) in deep learning-based medical image analysis,

    B. H. M. van der Velden, H. J. Kuijf, K. G. A. Gilhuijs, and M. A. Viergever, “Explainable artificial intelligence (XAI) in deep learning-based medical image analysis,” Medical Image Analysis, vol. 79, p. 102470, 2022

    2022

  9. [17]

    Re- solving challenges in deep learning-based analyses of histopathological images using explanation methods,

    M. H ¨agele, P. Seegerer, S. Lapuschkinet al., “Re- solving challenges in deep learning-based analyses of histopathological images using explanation methods,” Scientific Reports, vol. 10, no. 1, p. 6423, 2020

    2020

  10. [18]

    A survey on explainable artificial intelligence (XAI) techniques for visualizing deep learning models in medical imaging,

    D. Bhati, F. Neha, and M. Amiruzzaman, “A survey on explainable artificial intelligence (XAI) techniques for visualizing deep learning models in medical imaging,” Journal of Imaging, vol. 10, no. 10, p. 239, 2024

    2024

  11. [19]

    Explainabil- ity and causability in digital pathology,

    M. Plass, M. Kargl, T.-R. Kiehlet al., “Explainabil- ity and causability in digital pathology,”The Journal of Pathology: Clinical Research, vol. 9, no. 4, pp. 251– 260, 2023

    2023

  12. [20]

    Samek, G

    W. Samek, G. Montavon, A. Vedaldiet al.,Explain- able AI: Interpreting, Explaining and Visualizing Deep Learning, ser. Lecture Notes in Computer Science. Springer, 2019, vol. 11700

    2019

  13. [21]

    Shedding light on the black box of a neural network used to de- tect prostate cancer in whole slide images by occlusion- based explainability,

    M. Gallo, V . Kraj ˇnansk´y, R. Nenutilet al., “Shedding light on the black box of a neural network used to de- tect prostate cancer in whole slide images by occlusion- based explainability,”N. Biotechnol., vol. 78, pp. 52–67, oct 2023

    2023

  14. [22]

    Grad- CAM: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Daset al., “Grad- CAM: Visual explanations from deep networks via gradient-based localization,”International Journal of Computer Vision, vol. 128, no. 2, p. 336–359, Oct. 2019

    2019

  15. [23]

    Use hirescam instead of grad-cam for faithful explanations of convolu- tional neural networks,

    R. L. Draelos and L. Carin, “Use hirescam instead of grad-cam for faithful explanations of convolu- tional neural networks,” 2021. [Online]. Available: https://arxiv.org/abs/2011.08891

    2021 arXiv

  16. [24]

    Montavon, A

    G. Montavon, A. Binder, S. Lapuschkinet al.,Layer- Wise Relevance Propagation: An Overview. Cham: Springer International Publishing, 2019, pp. 193–209

    2019

  17. [25]

    Deep Feature Factorization for Concept Discovery,

    E. Collins, R. Achanta, and S. S ¨usstrunk, “Deep Feature Factorization for Concept Discovery,” inComputer Vi- sion – ECCV 2018, V . Ferrari, M. Hebert, C. Sminchis- escu, and Y . Weiss, Eds. Cham: Springer International Publishing, 2018, vol. 11218, pp. 352–368, series Title:...

    2018

  18. [26]

    Segmentation by factoriza- tion: Unsupervised semantic segmentation for pathol- ogy by factorizing foundation model features,

    J. Gildenblat and O. Hadar, “Segmentation by factoriza- tion: Unsupervised semantic segmentation for pathol- ogy by factorizing foundation model features,” 2024. [Online]. Available: https://arxiv.org/abs/2409.05697

    2024 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.