REVIEW 5 major objections 6 minor 1 cited by
A Methodological Framework for Measuring Spatial Labeling Similarity
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read SLAM compares two spatial labelings by turning them into weighted graphs, extracting edge-type distributions, and measuring distributional discrepancy with sliced Wasserstein distances and maximum mean discrepancy.
desk verdict SLAM is a genuine new metric for spatial labeling similarity with a clear framework and thoughtful experiments, but its severity validation is circular and its empirical support is thin; worth a serious look, not yet benchmark-ready. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Extended reading notes
Core claim
The central claim is that SLAM 'provides a comprehensive and accurate reflection of labeling quality' for spatial labeling results in spatial transcriptomics 'as per their similarity with ground truth labeling' (Abstract, Section 5.4). This means SLAM is the only benchmarked metric that simultaneously accounts for label agreement, topology of label distribution, and heterogeneous mismatch severity, and in the real-data case it is the only metric that ranks GraphST above STAGATE.
Load-bearing premise
The edge-weight function in Eq. (9) assumes that gene-expression similarity Sim(x_u, x_v) is a valid monotone proxy for mislabel severity: similar spots of the same type should be penalized more when assigned different labels, and dissimilar spots of different types should be penalized more when assigned the same label. If expression similarity does not track biological or clinical severity for a given tissue or labeling task, SLAM's severity adjustment is not unbiased and its claimed advantage over ARI and NMI collapses. This assumption enters in Step II (Section 3.2) and drives the positive Q coefficients in Cases V and VI.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a four-step methodological framework for measuring similarity between two spatial labelings: label matching, construction of a label-conditional attributed graph, extraction of graph-attribute distributions, and computation of a distributional discrepancy. The framework is instantiated as SLAM for evaluating spatial transcriptomics domain labels against ground truth, combining label agreement, spatial topology, and gene-expression-derived mismatch severity through edge weights, KDE, and a sliced-Wasserstein/MMD discrepancy. Seven experimental cases compare SLAM with fourteen benchmark metrics using a Q coefficient that captures consistency and sensitivity, and the authors report that SLAM is the only metric with positive Q across the simulated severity cases and the only metric that ranks GraphST above STAGATE on a real breast cancer tissue section.
Significance. SLAM addresses a genuine gap: no widely used ST clustering evaluation metric jointly accounts for spatial topology and heterogeneous mismatch severity, and the modular framework is clearly structured. The paper provides a concrete implementation, public code, and experimental cases that isolate distinct failure modes, which is a strength. However, the central claim that SLAM provides a comprehensive and accurate reflection of labeling quality is currently supported mainly by validation designs that feed the same gene-expression similarity into both the metric and the expected answer, and by single synthetic realizations without error bars. If the severity model can be validated against an independent ground truth and the kernel-theoretic proof is corrected, the method would be a useful contribution to spatial clustering evaluation.
major comments (5)
- [Section 5.4, Table 2, Figure 7] The validation of SLAM's severity adjustment is circular. In Eq. (9), the edge weights are W = Sim(x_u, x_v) on same-ground-truth-label edges and W = 1 - Sim(x_u, x_v) on cross-label edges, so the severity model is entirely a monotone transform of gene-expression cosine similarity. Case VI then defines the expected severity ranking using exactly the same gene-expression similarity (average cosine 0.791 versus 0.673 reported in Section 5.3), so SLAM cannot fail Case VI by construction and the positive Q in Table 1 provides no independent evidence for the severity-encoding claim. Case V similarly assumes clinical false-negative severity without an independent severity measure. Please validate the severity model against a severity ground truth that is not a function of Sim (for example, histopathologic or clinical outcome labels), or state explicitly that severity encoding is an untested modeling assumption.
- [Table 1, Eq. (32)] All Q coefficients in Table 1 are point estimates from a single synthetic realization per case, with no replicates, confidence intervals, or statistical tests. Because the central claim is that SLAM is the only metric with positive Q in every case, a single point estimate is insufficient support, especially since several Q values are small (0.073-0.257). Please report distributions of Q over repeated random realizations with different spot configurations, mislabel locations, and sampling of the data, together with bootstrap confidence intervals or paired tests.
- [Appendix A.2, Eqs. (23)-(29)] The proof that the sliced Wasserstein distance is conditionally negative definite is not valid as written. In Eq. (23), h_{P→Q} is defined with q(x) in the integrand, whereas the 2-Wasserstein distance is ∫(T_{P→Q}(x) - x)^2 p(x) dx, so the claimed equality W2(P,Q) = ⟨h_{P→Q}, h_{P→Q}⟩ does not hold with the given definition. In Eq. (27), the substitution from W2(P, Π) to an expression involving T_{Π→Q} appears to invert the direction of transport maps (T_{P→Q} maps P to Q, so T_{P→Q}(u) is not generally defined for u in the support of Q). Since the positive definiteness of the kernel Ξ in Eq. (15), the MMD representation in Eq. (13), and the claimed range d ∈ [0,2] all depend on this property, the theoretical foundation needs a corrected proof or a precise citation to a known theorem.
- [Section 5.4, Table 2, Figure 7] The real-data claim that SLAM is the only metric that ranks GraphST above STAGATE rests on a single tissue section and on the authors' visual assessment that GraphST is better because STAGATE produces false negatives in one encircled region. There is no independent reference standard for mismatch severity in this slice, no replication across slices or datasets, and no blinded or quantitative criterion for the visual ranking. Please provide a quantitative severity scoring protocol and evaluate on multiple spatial transcriptomics datasets or sections.
- [Appendix E, Eq. (15)] The sensitivity analysis covers only the KDE bandwidth h, leaving the other free parameters unexamined: the Gaussian kernel bandwidth γ in Eq. (15), the mutual k-nearest-neighbor count k in Eq. (6), and the number of sliced Wasserstein projection directions. Given that the reported Q values are small, the metric's qualitative behavior in Cases V and VI could depend on these choices. Please report sensitivity of the Q coefficients to these parameters, or justify fixed defaults empirically.
minor comments (6)
- [Section 3.2 and Algorithm 1] The main text defines the graph via mutual k-nearest neighbors in Eq. (6), while Algorithm 1 (lines 5-6) additionally introduces a fixed-radius edge-weight set W_s and sets it to 1; these two edge definitions should be reconciled.
- [Eq. (9)] The function Sim(x_u, x_v) is left unspecified in the main text; Case VI mentions normalized cosine similarity, but a formal definition, including the normalization and handling of zero-expression genes, should appear next to Eq. (9).
- [Appendix E] Appendix E refers to 'case VI (Section C.1)', but Case VI is presented in Section 5.3 and Section C.1 contains Case II; the cross-reference should be corrected.
- [Eq. (32)] The symbol n is used both for the number of spots and for the sign indicator in the Q coefficient; please use a different symbol for the sign indicator in Eqs. (32)-(34).
- [Eqs. (13)-(14)] The text says n0 and n1 are 'the number of sampled distributions', but they are the sample sizes of edge-attribute vectors; please rephrase to avoid confusion.
- [Appendix C.1] Case II reports the trend of SLAM and error rate only qualitatively; a correlation coefficient or rank correlation between SLAM and error rate across the ten configurations would make the monotonicity claim quantitative.
Assumptions & free parameters
free parameters (4)
- KDE bandwidth h =
0.1 (default)
- sliced Wasserstein Gaussian kernel bandwidth gamma (psi) =
not specified
- mutual kNN neighbor count k =
not specified
- number of sliced Wasserstein projection directions =
not specified
assumptions (4)
- domain assumption Ground truth labels Y^(0) are available and treated as correct.
- domain assumption A mutual k-nearest-neighbor graph built from spot locations adequately encodes the spatial organization of tissue domains.
- ad hoc to paper Gene-expression cosine similarity is a valid monotone measure of biological similarity and mislabel severity.
- standard math Squared sliced Wasserstein distance is conditionally negative definite and the Gaussian kernel induces a unique RKHS.
Cite this review
Pith. "Pith review of A Methodological Framework for Measuring Spatial Labeling Similarity." pith.science (2026). https://pith.science/paper/5VNPOSQY
@misc{pith2026250514128,
author = {Pith},
title = {Pith review of: A Methodological Framework for Measuring Spatial Labeling Similarity},
year = {2026},
howpublished = {\url{https://pith.science/paper/5VNPOSQY}},
note = {Machine review of arXiv:2505.14128}
}
read the original abstract
Spatial labeling assigns labels to specific spatial locations to characterize their spatial properties and relationships, with broad applications in scientific research and practice. Measuring the similarity between two spatial labelings is essential for understanding their differences and the contributing factors, such as changes in location properties or labeling methods. An adequate and unbiased measurement of spatial labeling similarity should consider the number of matched labels (label agreement), the topology of spatial label distribution, and the heterogeneous impacts of mismatched labels. However, existing methods often fail to account for all these aspects. To address this gap, we propose a methodological framework to guide the development of methods that meet these requirements. Given two spatial labelings, the framework transforms them into graphs based on location organization, labels, and attributes (e.g., location significance). The distributions of their graph attributes are then extracted, enabling an efficient computation of distributional discrepancy to reflect the dissimilarity level between the two labelings. We further provide a concrete implementation of this framework, termed Spatial Labeling Analogy Metric (SLAM), along with an analysis of its theoretical foundation, for evaluating spatial labeling results in spatial transcriptomics (ST) \textit{as per} their similarity with ground truth labeling. Through a series of carefully designed experimental cases involving both simulated and real ST data, we demonstrate that SLAM provides a comprehensive and accurate reflection of labeling quality compared to other well-established evaluation metrics. Our code is available at https://github.com/YihDu/SLAM.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Learning Optimal Multimodal Information Bottleneck Representations
OMIB introduces a bound on the MIB redundancy weight and per-modality adaptive weighting, but the proof of the bound uses a different objective and unjustified independence.
Reference graph
Works this paper leans on
-
[1]
[Alexandrov and Bartels, 2013] Theodore Alexandrov and Andreas Bartels. Testing for presence of known and un- known molecules in imaging mass spectrometry.Bioin- formatics, 29(18):2335–2342,
work page 2013
-
[6]
[Brenier, 1991] Yann Brenier. Polar factorization and mono- tone rearrangement of vector-valued functions.Communi- cations on pure and applied mathematics, 44(4):375–417,
work page 1991
-
[15]
Exploring network structure, dynam- ics, and function using networkx
[Hagberget al., 2008 ] Aric Hagberg, Pieter J Swart, and Daniel A Schult. Exploring network structure, dynam- ics, and function using networkx. Technical report, Los Alamos National Laboratory (LANL), Los Alamos, NM (United States),
work page 2008
-
[18]
A short tutorial on the weisfeiler-lehman test and its variants
[Huang and Villar, 2021] Ningyuan Teresa Huang and Soledad Villar. A short tutorial on the weisfeiler-lehman test and its variants. InICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8533–8537. IEEE,
work page 2021
-
[19]
[Jolliffe and Stephenson, 2012] Ian T Jolliffe and David B Stephenson.Forecast verification: a practitioner’s guide in atmospheric science. John Wiley & Sons,
work page 2012
-
[20]
Sliced wasserstein kernels for probabil- ity distributions
[Kolouriet al., 2016 ] Soheil Kolouri, Yang Zou, and Gus- tavo K Rohde. Sliced wasserstein kernels for probabil- ity distributions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5258– 5267,
work page 2016
-
[25]
[Rand, 1971] William M Rand. Objective criteria for the evaluation of clustering methods.Journal of the Ameri- can Statistical association, 66(336):846–850,
work page 1971
-
[27]
[Rousseeuw, 1987] Peter J Rousseeuw. Silhouettes: a graph- ical aid to the interpretation and validation of cluster anal- ysis.Journal of computational and applied mathematics, 20:53–65,
work page 1987
Show all 36 references
-
[29]
An irregular spatial cluster detection combining the genetic algorithm
[Wanget al., 2019 ] Tao Wang, Yitong Zhao, Yonglin Lei, Mei Yang, Shan Mei, et al. An irregular spatial cluster detection combining the genetic algorithm. InCS & IT Conference Proceedings, volume
2019
-
[34]
Recall, precision and average preci- sion.Department of Statistics and Actuarial Science, Uni- versity of Waterloo, Waterloo, 2(30):6,
[Zhu, 2004] Mu Zhu. Recall, precision and average preci- sion.Department of Statistics and Actuarial Science, Uni- versity of Waterloo, Waterloo, 2(30):6,
2004
-
[36]
A kernelk(s i, sj) = exp(−ψf(s i, sj)), ψ >0is positive definite⇐ ⇒fis conditionally negative definite
A Methodological Framework for Measuring Spatial Labeling Similarity Appendix The appendix is organized as follows: InAppendix A, we provide more technical details of the framework and implemen- tation, including the label matching functionM, and the theoretical foundation of ...
2015
-
[1971]
V-measure: A conditional entropy-based external cluster evaluation measure
[Rosenberg and Hirschberg, 2007] Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. InProceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language...
2007
-
[1979]
Deciphering spatial domains from spatially resolved tran- scriptomics with an adaptive graph attention auto-encoder
[Dong and Zhang, 2022] Kangning Dong and Shihua Zhang. Deciphering spatial domains from spatially resolved tran- scriptomics with an adaptive graph attention auto-encoder. Nature communications, 13(1):1739,
2022
-
[1984]
Sliced and radon wasserstein barycenters of measures.Journal of Mathematical Imaging and Vision, 51:22–45,
[Bonneelet al., 2015 ] Nicolas Bonneel, Julien Rabin, Gabriel Peyr ´e, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures.Journal of Mathematical Imaging and Vision, 51:22–45,
2015
-
[1987]
Spa- tially aware dimension reduction for spatial transcrip- tomics.Nature communications, 13(1):7203,
[Shang and Zhou, 2022] Lulu Shang and Xiang Zhou. Spa- tially aware dimension reduction for spatial transcrip- tomics.Nature communications, 13(1):7203,
2022
-
[1990]
Domain adaptive and fine-grained anomaly detection for single-cell sequencing data and beyond.arXiv preprint arXiv:2404.17454,
[Xuet al., 2024a ] Kaichen Xu, Yueyang Ding, Suyang Hou, Weiqiang Zhan, Nisang Chen, Jun Wang, and Xiaobo Sun. Domain adaptive and fine-grained anomaly detection for single-cell sequencing data and beyond.arXiv preprint arXiv:2404.17454,
-
[1991]
[Chung, 1997] Fan RK Chung.Spectral graph theory, vol- ume
1997
-
[1993]
The dynamic hungarian algorithm for the assignment problem with changing costs
[Mills-Tetteyet al., 2007 ] G Ayorkor Mills-Tettey, Anthony Stentz, and M Bernardine Dias. The dynamic hungarian algorithm for the assignment problem with changing costs. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR- 07-27,
2007
-
[1997]
John Wiley & Sons,
[Cover, 1999] Thomas M Cover.Elements of information theory. John Wiley & Sons,
1999
-
[1998]
Construction of permuta- tion tests.Journal of the American Statistical Association, 85(411):693–698,
[Welch, 1990] William J Welch. Construction of permuta- tion tests.Journal of the American Statistical Association, 85(411):693–698,
1990
-
[1999]
A cluster separation measure.IEEE transactions on pattern analysis and machine intelligence, (2):224– 227,
[Davies and Bouldin, 1979] David L Davies and Donald W Bouldin. A cluster separation measure.IEEE transactions on pattern analysis and machine intelligence, (2):224– 227,
1979
-
[2001]
[Huet al., 2021 ] Jian Hu, Xiangjie Li, Kyle Coleman, Amelia Schroeder, Nan Ma, David J Irwin, Edward B Lee, Russell T Shinohara, and Mingyao Li. Spagcn: Integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by gra...
2021
-
[2004]
const: an interpretable multi-modal contrastive learning framework for spatial transcriptomics.” biorxiv
[Zonget al., 2022 ] Yongshuo Zong, Tingyang Yu, Xuesong Wang, Yixuan Wang, Zhihang Hu, and Yu Li. const: an interpretable multi-modal contrastive learning framework for spatial transcriptomics.” biorxiv
2022
-
[2005]
A kernel two-sample test.The Journal of Machine Learning Research, 13(1):723–773,
[Grettonet al., 2012 ] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch ¨olkopf, and Alexander Smola. A kernel two-sample test.The Journal of Machine Learning Research, 13(1):723–773,
2012
-
[2007]
Wasserstein barycenter and its ap- plication to texture mixing
[Rabinet al., 2012 ] Julien Rabin, Gabriel Peyr ´e, Julie De- lon, and Marc Bernot. Wasserstein barycenter and its ap- plication to texture mixing. InScale Space and Variational Methods in Computer Vision: Third International Confer- ence, SSVM 2011, Ein-Gedi, Israel, May 29–J...
2012
-
[2008]
On clustering validation tech- niques.Journal of intelligent information systems, 17:107–145,
[Halkidiet al., 2001 ] Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis. On clustering validation tech- niques.Journal of intelligent information systems, 17:107–145,
2001
-
[2012]
Data filtering and its prioritization in pipelines for spatial segmentation of mass spectrometry imaging.Analytical chemistry, 93(11):4788–4793,
[Guoet al., 2021 ] Lei Guo, Zhenxing Hu, Chao Zhao, Xi- angnan Xu, Shujuan Wang, Jingjing Xu, Jiyang Dong, and Zongwei Cai. Data filtering and its prioritization in pipelines for spatial segmentation of mass spectrometry imaging.Analytical chemistry, 93(11):4788–4793,
2021
-
[2013]
Spatial deconvolution of her2- positive breast tumors reveals novel intercellular relation- ships.bioRxiv, pages 2020–07,
[Anderssonet al., 2020 ] Alma Andersson, Ludvig Larsson, Linnea Stenbeck, Fredrik Salm ´en, Anna Ehinger, Sunny Wu, Ghamdan Al-Eryani, Daniel Roden, Alex Swar- brick, ˚Ake Borg, et al. Spatial deconvolution of her2- positive breast tumors reveals novel intercellular relation- ...
2020
-
[2015]
Mars: discovering novel cell types across heterogeneous single-cell experiments.Na- ture methods, 17(12):1200–1206,
[Brbi´cet al., 2020 ] Maria Brbi ´c, Marinka Zitnik, Sheng Wang, Angela O Pisco, Russ B Altman, Spyros Darma- nis, and Jure Leskovec. Mars: discovering novel cell types across heterogeneous single-cell experiments.Na- ture methods, 17(12):1200–1206,
2020
-
[2016]
Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with graphst
[Longet al., 2023 ] Yahui Long, Kok Siong Ang, Mengwei Li, Kian Long Kelvin Chong, Raman Sethi, Chengwei Zhong, Hang Xu, Zhiwei Ong, Karishma Sachaphibulkij, Ao Chen, et al. Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with graphst. ...
2023
-
[2019]
Collective dynamics of ‘small-world’networks
[Watts and Strogatz, 1998] Duncan J Watts and Steven H Strogatz. Collective dynamics of ‘small-world’networks. nature, 393(6684):440–442,
1998
-
[2020]
Theory of reproduc- ing kernels.Transactions of the American mathematical society, 68(3):337–404,
[Aronszajn, 1950] Nachman Aronszajn. Theory of reproduc- ing kernels.Transactions of the American mathematical society, 68(3):337–404,
1950
-
[2021]
A probabilistic interpretation of precision, recall and f- score, with implication for evaluation
[Goutte and Gaussier, 2005] Cyril Goutte and Eric Gaussier. A probabilistic interpretation of precision, recall and f- score, with implication for evaluation. InEuropean con- ference on information retrieval, pages 345–359. Springer,
2005
-
[2022]
Advances in spatial transcriptomic data analysis.Genome research, 31(10):1706–1718,
[Drieset al., 2021 ] Ruben Dries, Jiaji Chen, Natalie Del Rossi, Mohammed Muzamil Khan, Adriana Sistig, and Guo-Cheng Yuan. Advances in spatial transcriptomic data analysis.Genome research, 31(10):1706–1718,
2021
-
[2023]
Accuracy measures: theoretical and practical concerns.International journal of forecasting, 9(4):527–529,
[Makridakis, 1993] Spyros Makridakis. Accuracy measures: theoretical and practical concerns.International journal of forecasting, 9(4):527–529,
1993
-
[2024]
Benchmarking spatial clustering methods with spatially resolved transcriptomics data.Nature Meth- ods, 21(4):712–722,
[Yuanet al., 2024 ] Zhiyuan Yuan, Fangyuan Zhao, Senlin Lin, Yu Zhao, Jianhua Yao, Yan Cui, Xiao-Yong Zhang, and Yi Zhao. Benchmarking spatial clustering methods with spatially resolved transcriptomics data.Nature Meth- ods, 21(4):712–722,
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.