REVIEW 3 major objections 4 minor 44 references
Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Pixel-wise heatmaps expose hidden biases in pathology AI
desk verdict A solid proof-of-concept for pixel-level LRP heatmaps in histopathology bias detection and cell-level evaluation; the faithfulness assumption needs an ablation but the core contribution holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is Layer-wise Relevance Propagation (LRP), which distributes the network's output score backward through the network so that every input pixel receives a relevance value indicating how strongly it speaks for or against the predicted class. The specific rules are the epsilon rule for the classifier layer, $$$R_i^{{(l)}}$ = \sum_j \frac{z_{ij}}{\sum_{i'} z_{i'j} + \varepsilon \cdot \operatorname{sign}(\sum_{i'} z_{i'j})} $R_j^{{(l+1)}}$,$$ and the $\alpha$-$\beta$ rule for convolutional layers, $$$R_i^{{(l)}}$ = \sum_j \left(\$\alpha$ \frac{z^+_{ij}}{\sum_{i'} z^+_{i'j}} + \$\beta$ \frac{z^-_{ij}}{\sum_{i'} z^-_{i'j}}\right) $R_j^{{(l+1)}}$,$$ with $\alpha=1,\beta=0$. These pixel-level maps carry the argument because they show where the model is looking; in the bias experiments the heatmaps, not the accuracy scores, are the evidence that the model exploited an artefact, the patch centre, or the absence of necrosis. The same maps feed the cell-level ROC evaluation, where each annotated cell receives the mean relevance in a circular neighbourhood.
What would settle it
Take a fresh model trained on the same tumour-discrimination task, plant a known hidden marker such as a small colour square in every cancer patch, and check whether LRP on a single held-out patch localises the marker before any biological structure is highlighted. A quantitative version: compare LRP heatmaps with a perturbation test in which the highest-relevance pixels are occluded and the prediction drop is measured; if predictions do not degrade when top-relevance regions are removed, the relevance scores are not faithfully ranking decision-contributing pixels.
Extended reading notes
Core claim
The central claim is that Layer-wise Relevance Propagation produces high-resolution heatmaps that make the classifier's decision process visible at cell resolution, and that this visibility resolves two practical problems in digital pathology. First, heatmaps reveal biases invisible to patch-level metrics: a model trained on patches labelled by the centre cell concentrates relevance at the patch centre, a model trained with a tiny 5x5-pixel corner artefact on every cancer patch learns the artefact while still scoring 100% patch accuracy, and a model trained without necrotic tissue assigns positive cancer relevance to necrosis at test time. In each case the bias is detectable on one or a few affected images without a large labelled hold-out set. Second, the heatmaps allow quantitative evaluation at the cellular level: ROC curves built from relevance at annotated cell locations beat zero-, one-, and random-heatmap baselines in all three tumour entities, and this granularity exposes behaviours patch-level F1 misses, such as positive relevance assigned to tumour-infiltrating lymphocytes in the SKCM model.
Load-bearing premise
The whole approach rests on the heatmaps faithfully reflecting what the network actually used for its decision; if LRP attribution does not track the true decision process, the bias detections, the cell-level ROC evaluation, and the interpretation of heatmap differences all lose their basis.
Editorial extensions
If this is right
- A model that learns the wrong thing can be caught from a single heatmap, without collecting a new annotated test set, which lowers the cost of bias detection in small medical datasets.
- Cell-level ROC can reveal failure modes invisible in patch-level F1; in the SKCM example, small positive relevance on tumour-infiltrating lymphocytes flattens the curve, and excluding those two tiles raises AUC to 94%.
- The balance of positive and negative relevance in a heatmap tracks the precision/recall trade-off induced by class sampling ratios, letting deployment teams choose a ratio for sensitivity- or specificity-oriented applications.
- Heatmaps support iterative dataset enrichment: samples where the model performs poorly on unlabelled data can be flagged and sent to experts for annotation, making the training distribution more comprehensive.
- The pixel-resolution advantage is not tied to LRP specifically; the paper argues that any fine-grained heatmap method should offer the same bias-detection and cell-evaluation benefits.
Reading between the lines
- Editorial inference: the spatial concentration of relevance, such as how much absolute relevance falls near a patch centre, could serve as a cheap, label-free screening metric for dataset bias before any test set is evaluated.
- Editorial inference: because the cell-level ROC protocol only scores relevance on annotated cells, it could be extended to measure relevance on surrounding stroma and lymphocytes, capturing the context features the paper acknowledges are important but does not quantify.
- Editorial inference: the same heatmap-first audit could be applied to whole-slide scanning pipelines, where staining variation and tissue folds are frequent, by averaging heatmaps per batch or per contributing site.
- Editorial inference: the faithfulness of LRP itself remains an open question; one could build a controlled benchmark with synthetic known features and compare different attribution methods, which the paper does not do.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using pixel-wise explanation heatmaps generated by Layer-wise Relevance Propagation (LRP) as a diagnostic instrument for deep learning models in digital histopathology. The authors fine-tune GoogLeNet classifiers for tumor versus non-tumor discrimination on H&E-stained TCGA patches from three entities (BRCA, SKCM, LUAD). They evaluate the heatmaps quantitatively at the cell level using ROC curves against independent pathologist annotations of cells, and they use heatmaps to investigate three types of dataset biases: a dataset-wide labeling bias (label determined by the patch center), a class-correlated bias (a 5x5 px artificial artefact inserted into all cancer patches), and a sampling bias (necrotic tissue excluded from training). They report that heatmaps reveal these biases, that the class-correlated bias is detectable on a single sample, and that counteracting the center bias with translation augmentation improves AUC by about 5%. The central claim is that pixel-wise heatmaps allow detecting and removing hidden biases and enable cell-level quantitative evaluation, offering advantages over patch-level metrics.
Significance. If the central claim holds, the paper demonstrates a practically useful role for explanation heatmaps in the development and deployment of pathology classifiers: cell-level evaluation without expensive detection models, and bias detection on single samples without large annotated test sets. The controlled class-correlated bias experiment (a known 5x5 px corruption) is clean and convincing as a proof of concept, and the cell-level ROC evaluation against board-certified pathologist annotations is a strong point; the authors also provide public code and data links and compare with Grad-CAM. However, the entire diagnostic claim rests on the assumption that LRP relevance scores faithfully reflect the features the model actually uses. This faithfulness is not independently validated for these models, and several supporting experiments are informal (no error bars, small numbers of tiles, a single acceptance radius). These gaps affect the strength of the paper's main conclusions and require further evidence before the claims can be accepted as established.
major comments (3)
- [Explaining classifier decisions; Quantitative evaluation of explanation heatmaps; Uncovering biases] The load-bearing assumption that LRP relevance faithfully reflects the model's decision evidence is unvalidated for these GoogLeNet pathology models. Equations (1) and (2) define the LRP rules, but no independent verification of faithfulness is provided. Consequently, both the cell-level ROC in 'Quantitative evaluation of explanation heatmaps' and the bias-detection experiments in 'Uncovering biases' inherit an unverified premise: positive relevance is interpreted as 'the feature the model relies on.' The controlled corruption experiment demonstrates that LRP highlights a label-correlated 5x5 px artefact, but this is a correlation, not a causal demonstration: it could in principle occur even if the model primarily used other features. To secure the central claim, the authors should add an intervention test (e.g., removing or recolouring the artefact and measuring the prediction change) or a second attribution method as a cross-check, on at least the class-correlated and sampling-bias experiments.
- [Uncovering biases (Dataset bias; Sampling bias); Supplemental Fig. 7] The quantitative support for the bias-removal claims is thin. The reported 5% AUC improvement after counteracting the center bias is described as a 'trend' and is presented without confidence intervals or significance testing; the underlying comparison uses a single test set of seven tiles. The sampling-bias evaluation in 'Sampling bias' is based on only five tiles and 'half of the considered regions' showing positive relevance, with no error bars or statistical assessment. These are small-sample, single-instance results, and the paper generalizes from them to the statement that heatmaps help 'remove the effects of common hidden biases, which improves generalization within and across datasets.' I ask the authors to either provide error bars and significance tests or soften the general claim to what the current evidence strictly supports.
- [Uncovering biases and Table 2] The claim that heatmaps reveal biases 'often without the necessity of labels' and 'on a single sample' is only demonstrated for biases that were already known to the experimenters: the center bias is known by construction, the class-correlated artefact is an artificial insertion, and the sampling bias is studied by comparing a model trained without necrosis against one trained with it. The paper does not provide a protocol for discovering an unknown bias in a blind setting, nor a quantitative criterion for what pattern in a heatmap constitutes evidence of bias. This limits the practical scope of the central claim. I suggest the authors either add a blind or semi-blind validation (e.g., presenting heatmaps without knowledge of the bias to an expert) or explicitly restrict the claim to 'known by construction' biases in the conclusion.
minor comments (4)
- [Conclusion] In the Conclusion, the sentence 'we can, both quantitatively and quantitatively, compare features' should read 'both qualitatively and quantitatively.'
- [Quantitative evaluation of explanation heatmaps] The choice of a single acceptance radius of ~50 px around each point annotation is not tested for sensitivity; a short analysis of how the cell-level ROC changes with the radius would strengthen the quantitative evaluation.
- [Uncovering biases (Dataset bias)] The dataset bias experiment uses a separate 2116-tile breast-cancer dataset, but this is not clearly distinguished from the TCGA-BRCA dataset described in Table 1; please clarify the relationship and avoid potential confusion.
- [Method (Convolutional neural network training)] The sentence 'For the training and test set patient cases were split 80/20, while keeping the ratio of healthy and diseased cases constant' would be clearer as 'keeping the ratio of healthy to diseased patients constant' or similar.
Circularity Check
No significant circularity: the paper's claims are benchmarked against independent expert annotations and known ground-truth corruptions, and its LRP parameters are imported, not fitted.
full rationale
The paper's central claim is that pixel-wise explanation heatmaps, specifically those generated by LRP, can detect biases and enable cell-level evaluation in histopathology. This is not circular: the cell-level ROC analysis is validated against independent, manually produced pathologist annotations of individual cells (Section 'Quantitative evaluation of explanation heatmaps'), and the bias-detection experiments use deliberately inserted, known ground-truth corruptions, such as the 5x5 px artefact correlated with the cancer class (Section '"Class-correlated" bias'). The LRP rules in Eqs. (1) and (2) and their parameter values are adopted from prior work (Montavon et al. [43]) rather than fitted to the pathology data, so the subsequent heatmaps are not constructed to match the evaluation labels. The paper explicitly acknowledges a limitation that only relevance on cells, not surrounding tissue, is measured in the ROC analysis (Section 'Limitations of visual explanations'), but this is an admitted scope restriction, not a circular reduction. Self-citations to LRP literature occur, but the load-bearing evidence is the external benchmark against expert labels and the controlled bias experiments, not the citations themselves. The assumption that LRP relevance faithfully reflects the model's decision process is a correctness or validity concern, not a circularity, because the paper's conclusions do not reduce by definition to that assumption; they are empirically testable against independent annotations. Therefore no circular step is exhibited.
Assumptions & free parameters
free parameters (2)
- LRP rule parameters (epsilon, alpha, beta) =
epsilon = 1, alpha = 1, beta = 0
- Cell relevance acceptance radius =
~50 px (half average cancer cell diameter)
assumptions (3)
- domain assumption LRP relevance scores faithfully reflect the classifier's decision process.
- domain assumption An optimal tumor classifier assigns positive relevance to all cancer cells, so cell-level ROC is a valid evaluation.
- domain assumption Necrosis is not relevant to tumor discrimination, so a model responding to necrosis has a sampling bias.
Cite this review
Pith. "Pith review of Resolving challenges in deep learning-based analyses of histopathological images using explanation methods." pith.science (2026). https://pith.science/paper/J562OVQR
@misc{pith2026190806943,
author = {Pith},
title = {Pith review of: Resolving challenges in deep learning-based analyses of histopathological images using explanation methods},
year = {2026},
howpublished = {\url{https://pith.science/paper/J562OVQR}},
note = {Machine review of arXiv:1908.06943}
}
read the original abstract
Deep learning has recently gained popularity in digital pathology due to its high prediction quality. However, the medical domain requires explanation and insight for a better understanding beyond standard quantitative performance evaluation. Recently, explanation methods have emerged, which are so far still rarely used in medicine. This work shows their application to generate heatmaps that allow to resolve common challenges encountered in deep learning-based digital histopathology analyses. These challenges comprise biases typically inherent to histopathology data. We study binary classification tasks of tumor tissue discrimination in publicly available haematoxylin and eosin slides of various tumor entities and investigate three types of biases: (1) biases which affect the entire dataset, (2) biases which are by chance correlated with class labels and (3) sampling biases. While standard analyses focus on patch-level evaluation, we advocate pixel-wise heatmaps, which offer a more precise and versatile diagnostic instrument and furthermore help to reveal biases in the data. This insight is shown to not only detect but also to be helpful to remove the effects of common hidden biases, which improves generalization within and across datasets. For example, we could see a trend of improved area under the receiver operating characteristic curve by 5% when reducing a labeling bias. Explanation techniques are thus demonstrated to be a helpful and highly relevant tool for the development and the deployment phases within the life cycle of real-world applications in digital pathology.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
LeCun, Y ., Bengio, Y . & Hinton, G. E. Deep learning.Nature 521, 436–444 (2015)
work page 2015
-
[2]
Deep learning in neural networks: An overview
Schmidhuber, J. Deep learning in neural networks: An overview. Neural Networks 61, 85–117 (2015)
work page 2015
-
[3]
Szegedy, C. et al. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 1–9 (2015)
work page 2015
-
[4]
Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 1097–1105 (2012)
work page 2012
-
[5]
Litjens, G. et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci. Rep. 6, 26286 (2016). 11
work page 2016
-
[6]
Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Analysis 42, 60–88 (2017)
work page 2017
- [7]
-
[8]
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017)
work page 2017
Show all 44 references
-
[9]
& Samek, W
Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R. & Samek, W. Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2912–2920 (2016)
2016
-
[10]
Lapuschkin, S. et al. Unmasking clever hans predictors and assessing what machines really learn. Nat. Commun. 10, 1096 (2019)
2019
-
[11]
& Zisserman, A
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
2013 arXiv
-
[12]
Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision – ECCV 2014, 818–833 (2014)
2014
-
[13]
M., Fuchs, T
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J. & Lipson, H. Understanding neural networks through deep visualization. arXiv preprint arXiv: 1506.06579 (2015)
2015 arXiv
-
[14]
Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10, 1–46 (2015)
2015
-
[15]
Selvaraju, R. R. et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 618–626 (2017)
2017
-
[16]
T., Alber, M., Müller, K.-R
Kindermans, P., Schütt, K. T., Alber, M., Müller, K.-R. & Dähne, S. PatternNet and PatternLRP - improving the interpretability of neural networks. stat 1050, 16 (2017)
2017
-
[17]
& Müller, K.-R
Montavon, G., Bach, S., Binder, A., Samek, W. & Müller, K.-R. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognit. 65, 211–222 (2017)
2017
-
[18]
M., Cohen, T
Zintgraf, L. M., Cohen, T. S., Adel, T. & Welling, M. Visualizing deep neural network decisions: Prediction difference analysis. In The fifth International Conference on Learning Representations (ICLR) (2017)
2017
-
[19]
Binder, A. et al. Towards computational fluorescence microscopy: Machine learning-based integrated prediction of morphological and molecular tumor profiles. arXiv preprint arXiv:1805.11178 (2018)
2018 arXiv
-
[20]
Korbar, B. et al. Looking under the hood: Deep neural network visualization to interpret whole-slide image analysis outcomes for colorectal polyps. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 821–827 (2017)
2017
-
[21]
Fuchs, T. J. & Buhmann, J. M. Computational pathology: Challenges and promises for tissue analysis. Comput. Med. Imaging Graph. 35, 515–530 (2011)
2011
-
[22]
Holzinger, A. et al. Towards the augmented pathologist : Challenges of explainable-ai in digital pathology. arXiv preprint arXiv:1712.06657 1–34 (2017)
2017 arXiv
-
[23]
Cruz-Roa, A., Caicedo, J. C. & González, F. A. Visual pattern mining in histology image collections using bag of features. Artif. Intell. Medicine 52, 91–106 (2011)
2011
-
[24]
& Tsai, C.-F
Huang, M.-W., Chen, C.-W., Lin, W.-C., Ke, S.-W. & Tsai, C.-F. Svm and svm ensembles in breast cancer prediction. PLOS ONE 12, 1–14 (2017)
2017
-
[25]
Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling
Yuan, Y .et al. Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling. Sci. Transl. Med. 4, 157–161 (2012)
2012
-
[26]
Sirinukunwattana, K. et al. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imaging 35, 1196–1206 (2016)
2016
-
[27]
Xu, J. et al. Stacked sparse autoencoder for nuclei detection on breast cancer histopathology images. IEEE Trans. Med. Imaging 35, 119–130 (2016)
2016
-
[28]
& Sun, J
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 770–778 (2016)
2016
-
[29]
Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features
Xu, Y .et al. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinforma. (2017). 12
2017
-
[30]
& Beck, A
Wang, D., Khosla, A., Gargeya, R., Irshad, H. & Beck, A. H. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718 (2016)
2016 arXiv
-
[31]
& Hajirasouliha, I
Khosravi, P., Kazemi, E., Imielinski, M., Elemento, O. & Hajirasouliha, I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine 27, 317–328 (2018)
2018
-
[32]
Bejnordi, B. E. et al. Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J. Med. Imaging 4, 044504 (2017)
2017
-
[33]
Alber, M. et al. Innvestigate neural networks! J. Mach. Learn. Res. 20, 1–8 (2019)
2019
-
[34]
A., Ovalle, J
Cruz-Roa, A. A., Ovalle, J. E. A., Madabhushi, A. & Osorio, F. A. G. A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In International Conference on Medical image computing and computer-assisted...
2013
-
[35]
Klauschen, F. et al. Scoring of tumor-infiltrating lymphocytes: From visual estimation to machine learning. In Seminars in cancer biology, vol. 52, 151–157 (2018)
2018
-
[36]
& Müller, H
Graziani, M., Andrearczyk, V . & Müller, H. Regression concept vectors for bidirectional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications, 124–132 (2018)
2018
-
[37]
Predicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P.et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. 115, E2970–E2979 (2018)
2018
-
[38]
Detecting cancer metastases on gigapixel pathology images
Liu, Y .et al. Detecting cancer metastases on gigapixel pathology images. arXiv preprint arXiv:1703.02442 (2017)
2017 arXiv
-
[39]
Coudray, N. et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nat. medicine 24, 1559 (2018)
2018
-
[40]
http://cancergenome.nih.gov
The cancer genome atlas. http://cancergenome.nih.gov
-
[41]
Caffe: Convolutional architecture for fast feature embedding
Jia, Y .et al. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia, 675–678 (2014)
2014
-
[42]
& Samek, W
Lapuschkin, S., Binder, A., Müller, K.-R. & Samek, W. Understanding and comparing deep neural networks for age and gender classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 1629–1638 (2017)
2017
-
[43]
& Müller, K.-R
Montavon, G., Samek, W. & Müller, K.-R. Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2017)
2017
-
[44]
biased" (left) and the
Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R. & Samek, W. The LRP toolbox for artificial neural networks. J. Mach. Learn. Res. 17, 3938–3942 (2016). Acknowledgements This work was supported by the German Ministry for Education and Research as Berlin Big Data Centre (0...
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.