{"id":"6fb24296-0694-4240-87d5-6ed845a18d99","arxiv_id":"2505.14128","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SLAM compares two spatial labelings by turning them into weighted graphs, extracting edge-type distributions, and measuring distributional discrepancy with sliced Wasserstein distances and maximum mean discrepancy.","lead":"Researchers propose a framework and a concrete metric, SLAM, for measuring how similar two spatial labelings are by comparing graphs built from locations, labels, and tissue attributes. The metric aims to judge spatial transcriptomics clustering results more fairly than standard scores by also accounting for where mismatches occur and how severe they are.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Severity validation is circular: Case VI defines mislabel severity via gene-expression similarity, which is exactly the Sim input to SLAM's Eq. (9); without an independent severity ground truth, the claimed advantage over ARI/NMI is unsubstantiated.","rationale":"The reader's weakest assumption points to the same load-bearing dependency: Eq. (9) assumes gene-expression similarity is a valid proxy for mislabel severity. My concern sharpens this into a circularity in the validation. In Case VI, severity is operationalized as expression dissimilarity, which is precisely the signal SLAM's edge weights consume; therefore the positive Q in Table 1 does not constitute independent evidence that SLAM measures heterogeneous mismatch severity. In Case V, clinical severity is assumed to be encoded by expression similarity without any supporting analysis. If this assumption is wrong, the claimed advantage over ARI/NMI in Cases V and VI collapses, and the 'comprehensive and accurate reflection' claim is not established. The paper does have independent support elsewhere: it releases code, includes carefully constructed topological simulations, and provides complexity and bandwidth sensitivity analyses. But none of these validate the severity component. The requested test—using an independent severity annotation and checking whether SLAM's rankings track it—would settle the concern. Since the reader's verdict is already CONDITIONAL and my concern reinforces rather than overturns it, the appropriate verdict remains CONDITIONAL.","tokens_in":18535,"tokens_out":7118,"duration_ms":70566,"concrete_test":"Obtain a real spatial transcriptomics dataset with an independent severity annotation that is not derived from expression cosine similarity—for example, pathologist-annotated distances from tumor margins or marker-gene-based lineage distances. Construct two labelings with identical label counts, identical topological structure, and equal numbers of mislabels, but with different severity profiles as in Cases V-VI. Compute SLAM and the Table 1 benchmarks, and compare the sign and magnitude of the Q coefficient against the independent severity ranking. If SLAM does not rank the independently-less-severe labeling as higher quality, or if replacing Sim in Eq. (9) with the independent severity matrix reverses the ranking, the severity-validity assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SLAM is the only benchmarked metric that correctly accounts for heterogeneous mismatch severity (Abstract; Section 5.3-5.4). That claim is not independently supported. In Eq. (9), SLAM's severity weights are W=Sim for same-ground-truth-label edges and W=1-Sim for cross-label edges, so the severity model is entirely a monotone transform of gene-expression similarity Sim(x_u, x_v). The only direct validation of this severity model is Case VI, where 'more severe' is defined as 'more dissimilar in gene expression' (Section 5.3: breast gland-adipose cosine similarity 0.791 vs breast gland-cancer 0.673). This is circular: the quantity used to define the expected answer is the same quantity fed into SLAM's weights. Case V likewise assumes, without independent evidence, that clinical false-negative severity is captured by gene-expression similarity between same-type spots. If Sim does not track biological or clinical severity in a given tissue, SLAM's severity adjustment is not a principled measure of mismatch severity, and its claimed advantage over ARI/NMI in Table 1 (positive Q in Cases V-VI) is not established. The real-data Case VII cannot break this circularity because it is scored by subjective visual inspection, not by an independent severity scale.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a four-step methodological framework for measuring similarity between two spatial labelings: label matching, construction of a label-conditional attributed graph, extraction of graph-attribute distributions, and computation of a distributional discrepancy. The framework is instantiated as SLAM for evaluating spatial transcriptomics domain labels against ground truth, combining label agreement, spatial topology, and gene-expression-derived mismatch severity through edge weights, KDE, and a sliced-Wasserstein/MMD discrepancy. Seven experimental cases compare SLAM with fourteen benchmark metrics using a Q coefficient that captures consistency and sensitivity, and the authors report that SLAM is the only metric with positive Q across the simulated severity cases and the only metric that ranks GraphST above STAGATE on a real breast cancer tissue section.","tokens_in":18788,"tokens_out":10212,"duration_ms":110757,"significance":"SLAM addresses a genuine gap: no widely used ST clustering evaluation metric jointly accounts for spatial topology and heterogeneous mismatch severity, and the modular framework is clearly structured. The paper provides a concrete implementation, public code, and experimental cases that isolate distinct failure modes, which is a strength. However, the central claim that SLAM provides a comprehensive and accurate reflection of labeling quality is currently supported mainly by validation designs that feed the same gene-expression similarity into both the metric and the expected answer, and by single synthetic realizations without error bars. If the severity model can be validated against an independent ground truth and the kernel-theoretic proof is corrected, the method would be a useful contribution to spatial clustering evaluation.","major_comments":[{"comment":"The validation of SLAM's severity adjustment is circular. In Eq. (9), the edge weights are W = Sim(x_u, x_v) on same-ground-truth-label edges and W = 1 - Sim(x_u, x_v) on cross-label edges, so the severity model is entirely a monotone transform of gene-expression cosine similarity. Case VI then defines the expected severity ranking using exactly the same gene-expression similarity (average cosine 0.791 versus 0.673 reported in Section 5.3), so SLAM cannot fail Case VI by construction and the positive Q in Table 1 provides no independent evidence for the severity-encoding claim. Case V similarly assumes clinical false-negative severity without an independent severity measure. Please validate the severity model against a severity ground truth that is not a function of Sim (for example, histopathologic or clinical outcome labels), or state explicitly that severity encoding is an untested modeling assumption.","section":"Section 5.4, Table 2, Figure 7"},{"comment":"All Q coefficients in Table 1 are point estimates from a single synthetic realization per case, with no replicates, confidence intervals, or statistical tests. Because the central claim is that SLAM is the only metric with positive Q in every case, a single point estimate is insufficient support, especially since several Q values are small (0.073-0.257). Please report distributions of Q over repeated random realizations with different spot configurations, mislabel locations, and sampling of the data, together with bootstrap confidence intervals or paired tests.","section":"Table 1, Eq. (32)"},{"comment":"The proof that the sliced Wasserstein distance is conditionally negative definite is not valid as written. In Eq. (23), h_{P→Q} is defined with q(x) in the integrand, whereas the 2-Wasserstein distance is ∫(T_{P→Q}(x) - x)^2 p(x) dx, so the claimed equality W2(P,Q) = ⟨h_{P→Q}, h_{P→Q}⟩ does not hold with the given definition. In Eq. (27), the substitution from W2(P, Π) to an expression involving T_{Π→Q} appears to invert the direction of transport maps (T_{P→Q} maps P to Q, so T_{P→Q}(u) is not generally defined for u in the support of Q). Since the positive definiteness of the kernel Ξ in Eq. (15), the MMD representation in Eq. (13), and the claimed range d ∈ [0,2] all depend on this property, the theoretical foundation needs a corrected proof or a precise citation to a known theorem.","section":"Appendix A.2, Eqs. (23)-(29)"},{"comment":"The real-data claim that SLAM is the only metric that ranks GraphST above STAGATE rests on a single tissue section and on the authors' visual assessment that GraphST is better because STAGATE produces false negatives in one encircled region. There is no independent reference standard for mismatch severity in this slice, no replication across slices or datasets, and no blinded or quantitative criterion for the visual ranking. Please provide a quantitative severity scoring protocol and evaluate on multiple spatial transcriptomics datasets or sections.","section":"Section 5.4, Table 2, Figure 7"},{"comment":"The sensitivity analysis covers only the KDE bandwidth h, leaving the other free parameters unexamined: the Gaussian kernel bandwidth γ in Eq. (15), the mutual k-nearest-neighbor count k in Eq. (6), and the number of sliced Wasserstein projection directions. Given that the reported Q values are small, the metric's qualitative behavior in Cases V and VI could depend on these choices. Please report sensitivity of the Q coefficients to these parameters, or justify fixed defaults empirically.","section":"Appendix E, Eq. (15)"}],"minor_comments":[{"comment":"The main text defines the graph via mutual k-nearest neighbors in Eq. (6), while Algorithm 1 (lines 5-6) additionally introduces a fixed-radius edge-weight set W_s and sets it to 1; these two edge definitions should be reconciled.","section":"Section 3.2 and Algorithm 1"},{"comment":"The function Sim(x_u, x_v) is left unspecified in the main text; Case VI mentions normalized cosine similarity, but a formal definition, including the normalization and handling of zero-expression genes, should appear next to Eq. (9).","section":"Eq. (9)"},{"comment":"Appendix E refers to 'case VI (Section C.1)', but Case VI is presented in Section 5.3 and Section C.1 contains Case II; the cross-reference should be corrected.","section":"Appendix E"},{"comment":"The symbol n is used both for the number of spots and for the sign indicator in the Q coefficient; please use a different symbol for the sign indicator in Eqs. (32)-(34).","section":"Eq. (32)"},{"comment":"The text says n0 and n1 are 'the number of sampled distributions', but they are the sample sizes of edge-attribute vectors; please rephrase to avoid confusion.","section":"Eqs. (13)-(14)"},{"comment":"Case II reports the trend of SLAM and error rate only qualitatively; a correlation coefficient or rank correlation between SLAM and error rate across the ten configurations would make the monotonicity claim quantitative.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The main risks are the circular validation of the severity model and the flawed proof of conditional negative definiteness in the appendix. Both are addressable in revision, but they are central to the paper's claims, so I would not recommend acceptance before they are fixed. The paper is otherwise within scope for a machine-learning or spatial-transcriptomics benchmarking venue; I have no competing interests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on 2505.14128. The real news is that SLAM is a genuine new metric, not a repackaging: it combines Jaccard-based label matching, graph edge typing, KDE, and MMD on sliced Wasserstein distances in a way I have not seen before, and the seven experimental cases are thoughtfully designed to separate label agreement, topology, and severity. The Q-coefficient summary is sensible, and the code being public is a plus. Credit where due: the paper shows that standard ARI/NMI ignore spatial organization and cannot see severity differences, and SLAM's construction addresses that gap.\n\nThe soft spots are real, though not fatal. The main one is a circularity in the severity validation. Case VI defines 'more severe' as 'more dissimilar in gene expression' — breast gland to adipose cosine 0.791 vs breast gland to cancer 0.673 — and then shows that SLAM, whose edge weights are built from exactly that Sim(xu, xv), gives a lower quality score to the cancer mislabel. The experiment demonstrates that SLAM reproduces its own severity model; it does not independently validate that gene-expression similarity is the right severity measure for biological or clinical decisions. Case V makes the same assumption for false negatives, and the real-data Case VII is scored by a single subjective visual inspection. So the headline claim that SLAM is 'comprehensive and accurate' is overreaching relative to the evidence.\n\nThree smaller issues: Q coefficients come from single synthetic realizations with no error bars or replicates; hyperparameters such as the kernel bandwidth, neighbor count, and number of projection directions are not all specified or sensitized (only the KDE bandwidth is); and the appendix proof of conditional negative definiteness has a suspicious optimal-transport composition step that should be repaired or replaced by a reference. None of these sink the paper, but they push it from 'benchmark-ready' to 'promising proposal needing revision.'\n\nWho gets value: people evaluating spatial clustering methods in spatial transcriptomics, and anyone building evaluation metrics that need to incorporate spatial topology. The metric itself is coherent, and the framework is reusable. I would send it to a serious referee rather than desk reject, because the core idea is worth engaging with and the gaps are fixable. I would not yet rely on it as the definitive benchmark.","headline":"SLAM is a genuine new metric for spatial labeling similarity with a clear framework and thoughtful experiments, but its severity validation is circular and its empirical support is thin; worth a serious look, not yet benchmark-ready.","tokens_in":19321,"tokens_out":4632,"would_cite":true,"duration_ms":49621,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:39:43.412705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}