Common Limitations of Image Processing Metrics: A Picture Story
read the original abstract
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation
An adaptation of Confident Learning detects directional label errors in segmentation datasets without clean ground truth and leverages encoder feature separability to mitigate bias and equalize performance across subgroups.
-
NucEval: A Robust Evaluation Framework for Nuclear Instance Segmentation
NucEval is a unified evaluation framework for nuclear instance segmentation that modifies standard metrics to handle vague regions, normalize scores, manage overlaps, and account for border uncertainty.
-
Current validation practice undermines surgical AI development
A multi-stage Delphi consensus with 92 experts catalogs widespread validation pitfalls in surgical AI video analysis across data, metrics, and reporting, supported by a systematic review and empirical experiments.
-
MONAI: An open-source framework for deep learning in healthcare
MONAI is a community-supported PyTorch framework that extends deep learning to medical data with domain-specific architectures, transforms, and deployment tools.
-
The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization
The autoPET3 challenge finds that leading AI models reach a mean Dice score of 0.66 for multitracer PET/CT lesion segmentation, with compositional generalization to unseen tracer-center pairs remaining an open problem...
-
The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization
The autoPET3 challenge finds good in-domain lesion segmentation performance in multitracer PET/CT but identifies compositional generalization to unseen tracer-center combinations as an open problem driven by volume ov...
-
Topology-Driven Fusion of nnU-Net and MedNeXt for Accurate Brain Tumor Segmentation on Sub-Saharan Africa Dataset
Pre-training nnU-Net and MedNeXt on BraTS 2025 data then fine-tuning on BraTS-Africa with added topology refinement yields NSD scores of 0.810, 0.829, and 0.895 for SNFH, NETC, and ET.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.