{"id":"b3dc3b0a-753b-40eb-a19c-c1d069db6720","arxiv_id":"2506.18668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-center benchmark of eight histopathology foundation models on skin cancer subtyping finds that VIRCHOW-2 performs best, while a new silhouette-based metric captures each model's center-related feature bias.","lead":"This preprint benchmarks eight AI foundation models for skin cancer subtyping from pathology slides, using two different classification pipelines on a multi-center dataset. It also introduces a new metric, the Foundation Model - Silhouette Index, to quantify how much each model's features encode the hospital or scanner that produced the image.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central FM-SI-to-accuracy trend rests on a non-significant correlation over eight models: the reported R² values (0.428, 0.346) correspond to p≈0.08 and p≈0.13, so the headline claim needs a significance test or tempering.","rationale":"I agree with the reader that FM-SI's t-SNE-based construction is a weak spot, but the more load-bearing issue is statistical: even taking FM-SI at face value, the central correlation is not significant. The paper's claim to 'show that extracting less biased features enhances classification performance' is the abstract's main takeaway and the basis for recommending FM-SI as a label-free model-selection tool. With n=8, R²=0.428 gives p≈0.08; this should be reported and tested. The concrete test would settle whether the relationship is robust. I do not think the paper should be rejected: the benchmark data, per-model accuracies, and FM-SI/RI comparison are useful and reproducible enough to warrant conditional acceptance pending significance/stability analysis and tempered claims.","tokens_in":9124,"tokens_out":5205,"duration_ms":53754,"concrete_test":"Compute a permutation test of the Spearman correlation between FM-SI and MI-SimpleShot BACC using the eight model-level points in Table S1 (shuffle FM-SI labels 10,000 times), plus leave-one-model-out R² and 95% bootstrap CIs. Also report the same for ABMIL. If p≥0.05 or the CI includes zero, the paper should present FM-SI as descriptive, not as a validated predictor of classification performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2's central claim is that lower FM-SI predicts higher balanced accuracy, especially for MI-SimpleShot, and the abstract generalizes this to model selection. The only quantitative support is Figure 4 and Table S1: linear R²=0.428 (MI-SimpleShot) and R²=0.346 (ABMIL) across 8 models. For n=8, these values correspond to F(1,6)=4.49 and 3.17, with p≈0.08 and p≈0.13 respectively, so neither trend reaches conventional significance. No confidence intervals, permutation tests, or leave-one-out analyses are reported, and one or two points (e.g., VIRCHOW-2 with high accuracy and mid-level FM-SI; CHIEF with low FM-SI but low accuracy) can change the inferred slope. Additionally, FM-SI is computed on all 621 slides, including the slides whose accuracy is measured, so it is not evaluated as an out-of-sample selector; the claimed predictive use 'without requiring extra labels' would require held-out estimation. The measurements themselves are not challenged; rather, the strength of the stated conclusion exceeds what a non-significant 8-point correlation can support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks eight histopathology foundation models as frozen patch-level feature extractors on the AI4SkIN multi-center whole-slide dataset for cutaneous spindle cell neoplasm subtyping, using two multiple-instance learning classifiers (ABMIL and the non-parametric MI-SimpleShot). It introduces the Foundation Model - Silhouette Index (FM-SI), computed by applying t-SNE to mean-pooled slide embeddings and taking the silhouette score with acquisition-center labels as clusters, to quantify center-related bias in the features. The main empirical claim is that models with lower FM-SI (less center-biased features) achieve higher balanced accuracy, with a stronger dependence for MI-SimpleShot (R2=0.428) than for ABMIL (R2=0.346). The paper also compares FM-SI with the previously published Robustness Index (|rho|=0.890) and reports that VIRCHOW-2 performs best in both classifiers, while CONCH and KEEP have the lowest FM-SI.","tokens_in":9536,"tokens_out":6446,"duration_ms":59091,"significance":"If the FM-SI-to-accuracy relationship holds, the paper would provide a label-free diagnostic for selecting robust histopathology feature extractors in multi-center settings, which is practically valuable for model deployment. The benchmark itself is useful: it covers eight FMs spanning self-supervised and vision-language pretraining, reports balanced accuracy over 5-fold stratified cross-validation with weighted cross-entropy, and transparently describes the MIL pipelines. The comparison against the established Robustness Index is a good validity check. However, the central statistical evidence is currently weak: the reported correlations are not significant at n=8, the FM-SI values depend on unstated t-SNE hyperparameters, and the metric is not evaluated out-of-sample. These issues are local and addressable; the dataset and protocol are described in enough detail that additional analyses can settle the claim.","major_comments":[{"comment":"The central claim that lower FM-SI predicts higher balanced accuracy is supported only by linear regressions over n=8 models (R2=0.428 for MI-SimpleShot and R2=0.346 for ABMIL). For n=8, these R2 values correspond to F(1,6)=4.49 and 3.17 (p approximately 0.08 and 0.13), so neither trend reaches conventional significance; no confidence intervals, permutation tests, or leave-one-model-out analyses are reported. Points such as VIRCHOW-2 (high BACC with mid-range FM-SI) and CHIEF (low FM-SI but low BACC for MI-SimpleShot) are visually influential. The abstract's statement that less biased features enhance classification performance should be tempered or supplemented with a significance test and an influence analysis.","section":"Section 4.2, Figure 4, Table S1"},{"comment":"FM-SI is computed as a silhouette score on a 2D t-SNE projection of mean-pooled slide embeddings. The paper does not report t-SNE hyperparameters (perplexity, learning rate, number of iterations, initialization) or random seeds, nor any stability analysis across t-SNE runs or across bootstrap resamples of slides. Because t-SNE is stochastic and locally focused, different projections could yield different FM-SI values (especially for KEEP at sigma=0.028 versus PLIP at sigma=0.686), and the correlation in Figure 4 depends on these values. Please report the exact t-SNE configuration, run multiple seeds, and show the resulting variation in FM-SI and in the R2 values; alternatively, compute the silhouette on the original high-dimensional embeddings as a robustness check.","section":"Section 2.2, Algorithm 1"},{"comment":"FM-SI is computed on all 621 slides (Algorithm 1 input D), the same slides whose balanced accuracy is measured in Table S1. Therefore the correlation in Figure 4 is descriptive and not an evaluation of FM-SI as a prospective model-selection criterion. The claim that FM-SI can select robust feature extractors 'without requiring extra labels' needs a held-out or leave-one-model-out analysis: for example, fit the regression on seven models and show that the eighth model's rank in FM-SI predicts its rank in BACC, or at least state explicitly that the current evidence is correlational.","section":"Section 4.2, evaluation protocol"}],"minor_comments":[{"comment":"The paper calls the contribution a 'novel benchmark', but the AI4SkIN dataset was published previously [3]; please clarify that the novelty lies in the evaluation protocol and the FM-SI metric, not the dataset itself.","section":"Abstract and Section 1"},{"comment":"The caption mentions solid and dotted lines for self-supervised and vision-language models, but the figure displays only points; either add the lines or correct the caption to describe point styles.","section":"Figure 4 caption"},{"comment":"MI-SimpleShot is cited to reference [2], the UNI Nature Medicine paper; please check whether this is the correct source for the method or replace it with the original MI-SimpleShot reference.","section":"Section 2.1 and Table S1"},{"comment":"The reported correlation |rho|=0.890 between FM-SI and RI is computed over the same eight models; please include the p-value and, ideally, a confidence interval.","section":"Section 4.1"},{"comment":"There are several typographical issues: 'feature representations extracted' repeats 'extracted', the t-SNE axes in Figure 3 read 'dimensión' with an accent, and 'Hospital Universitario San Secilio' is likely 'San Cecilio'. Please correct these throughout.","section":"Sections 2.2, 3.2, and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical study with a useful dataset and a clever label-free metric, but the headline correlation is not statistically significant with eight models, and the t-SNE dependence of FM-SI is unquantified. I believe the issues are fixable: add significance testing and influence analysis, report t-SNE hyperparameters and seed sensitivity, and perform a leave-one-model-out validation. I recommend major revision rather than rejection. A reproducibility concern: the paper provides no code or data-access link; for a benchmarking paper, sharing code would materially increase its value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a decent benchmark and the FM-SI metric is easy to like, but the central claim is currently over-claimed. The correlation between FM-SI and balanced accuracy over eight models (R²=0.428 for MI-SimpleShot, R²=0.346 for ABMIL) is not significant (p≈0.08 and p≈0.13 for n=8), so the abstract's statement that less biased features enhance classification performance is stronger than the data support.\n\nWhat's new: a multi-center benchmark of eight histopathology foundation models on the AI4SkIN skin cancer subtyping task, with two MIL classifiers, and a label-free metric for center shift. FM-SI is just t-SNE plus silhouette on slide-level mean-pooled embeddings, but applied to this problem it is a sensible check and it correlates strongly with the previously published Robustness Index (|ρ|=0.890), which is encouraging evidence that it measures something real. The experimental protocol is described well enough to reproduce approximately; using 5-fold stratified CV and weighted cross-entropy is fine, and the standard deviations in Table S1 are useful.\n\nSoft spots: the significance problem is the main one. With eight points and no confidence intervals, permutation tests, or leave-one-out checks, one model like VIRCHOW-2 (high accuracy, mid FM-SI) or CHIEF (low FM-SI, low accuracy) can change the slope. Second, FM-SI is computed on all 621 slides including the slides whose accuracy is used in the correlation, so it is not an out-of-sample selector despite the abstract implying it could be used 'without requiring extra labels'. Third, t-SNE is stochastic and the paper doesn't report perplexity, learning rate, or seeds; silhouette on a 2D embedding may not reflect high-dimensional structure, so the FM-SI ranking could be unstable. Fourth, no code is released. These are all addressable and don't invalidate the measurements themselves.\n\nWho this is for: anyone selecting a histopathology FM for a multi-center deployment, and the benchmark community. I'd want to see a revision that adds uncertainty estimates around FM-SI and the correlations, a held-out version of the metric, and a release of code and t-SNE settings. The paper deserves a serious referee rather than a desk reject. My own verdict would be conditional at best.","headline":"Useful multi-center FM benchmark and a plausible label-free center-bias metric, but the headline accuracy-vs-FM-SI relationship is non-significant over eight models and the metric's stability is unreported.","tokens_in":9921,"tokens_out":2432,"would_cite":false,"duration_ms":24745,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Histopathology foundation models that embed less acquisition-center information in their features achieve higher slide-level classification accuracy on a multi-center skin cancer subtyping task, and the effect is strongest for classifiers…","keywords":["histopathology foundation models","multiple instance learning","distribution shift","skin cancer subtyping","FM-SI","silhouette score","center bias","whole slide image"],"falsifier":"Recompute FM-SI for the eight models using multiple t-SNE random seeds and perplexity settings; if the model ranking changes materially (for example, KEEP no longer has the lowest score) or the $R^2$ values against balanced accuracy in Figure 4 fall to near zero, the claimed predictive relationship would not hold.","tokens_in":8919,"feed_emoji":"🔬","tokens_out":6367,"duration_ms":58449,"temperature":0.7,"pith_summary":"The paper tests whether a foundation model's tendency to encode the hospital or scanner a slide came from predicts how well that model will subtype skin cancers on whole slides. It introduces FM-SI, a label-free score computed by reducing mean-pooled slide embeddings to two dimensions and measuring how tightly slides cluster by acquisition center. Across eight histopathology foundation models and two classifier designs, the authors find a negative correlation between FM-SI and balanced accuracy: models with less center-biased features perform better, and the relationship is more pronounced for the non-parametric similarity-based classifier (MI-SimpleShot) than for attention-based MIL. If correct, FM-SI offers a way to choose a feature extractor for a new clinical task without collecting extra annotations, and it places the comparison of foundation models on a distribution-shift axis rather than accuracy alone.","feed_headline":"Center bias in AI features hurts skin cancer subtyping","feed_subtitle":"A label-free metric, FM-SI, predicts which histopathology models will generalize across hospitals.","key_machinery":"The load-bearing object is the Foundation Model - Silhouette Index (FM-SI), a label-free metric defined in Algorithm 1. It computes the slide-level embedding by averaging all patch embeddings from a frozen histopathology foundation model, projects these embeddings to two dimensions with t-SNE, and calculates the silhouette coefficient treating each acquisition center as a cluster. The resulting score quantifies how much of the encoder's feature space is organized around scanning site rather than tissue class, and the paper's main experiments correlate this score with balanced accuracy under two multiple-instance learning classifiers, ABMIL and MI-SimpleShot.","core_discovery":"The central discovery is an inverse relationship between the center-bias of a histopathology foundation model's features and its downstream slide-level classification performance in a six-class skin cancer subtyping benchmark built from 621 whole slide images digitized at two centers. Center bias is measured by the proposed Foundation Model - Silhouette Index (FM-SI): slide embeddings are obtained by mean-pooling the patch features, projected to two dimensions with t-SNE, and scored with the silhouette coefficient using acquisition center as the cluster label; values near 0 mean the features carry little center information, values near 1 mean the features cluster strongly by hospital. Across the eight models studied, FM-SI correlates with the existing Robustness Index with $|\\rho| = 0.89$, and FM-SI accounts for $R^2 = 0.428$ of the variance in MI-SimpleShot balanced accuracy and $R^2 = 0.346$ in ABMIL balanced accuracy. The paper also reports that attention-based MIL beats the non-parametric similarity classifier by on average 11.88 percentage points, and that the largest-scale model, VIRCHOW-2, reaches the top balanced accuracy (86.81% with ABMIL), while the vision-language models CONCH and KEEP produce the least center-biased features.","pith_inferences":["Beyond the reported results, FM-SI could serve as a drop-in encoder-selection criterion in deployment: compute it on a small multi-center validation set and pick the model with the lowest score.","Because FM-SI requires no class labels, it could be applied to other distribution shifts beyond scanner, such as staining protocol, tissue fixation, or demographic site, as long as shift labels are known.","The reported $R^2$ values are computed from only eight models, so an independent benchmark with more encoders and more centers would sharpen or weaken the linear trend.","The metric inherits sensitivity to t-SNE projection settings, so reporting stability across seeds and perplexities would make FM-SI a reproducible tool for model selection."],"forward_implications":["On the AI4SkIN multi-center skin cancer subtyping task, lower FM-SI predicts higher balanced accuracy, so FM-SI can rank candidate foundation models before any training of downstream classifiers.","The non-parametric MI-SimpleShot classifier is more sensitive to center bias than attention-based MIL, meaning label-free prototype classification inherits the quality of the feature extractor.","Vision-language pretraining (CONCH, KEEP) yields the least center-biased features, while the largest self-supervised corpus (VIRCHOW-2) gives the best accuracy despite moderate center bias.","Attention-based MIL outperforms MI-SimpleShot by about 12 percentage points on average across all models."],"supporting_citations":[{"why":"Supplies the MI-SimpleShot non-parametric MIL classifier and the UNI self-supervised foundation model used in the benchmark.","marker":"[2]"},{"why":"Provides the AI4SkIN multi-center dataset of 621 whole slide images with six cutaneous spindle cell neoplasm subtypes.","marker":"[3]"},{"why":"Supplies the attention-based MIL (ABMIL) framework that is the trainable classifier compared against MI-SimpleShot.","marker":"[6]"},{"why":"Defines the Robustness Index (RI) used as the external reference against which FM-SI is validated.","marker":"[7]"},{"why":"Provides the t-SNE dimensionality reduction that projects slide embeddings to two dimensions before the silhouette score is computed.","marker":"[10]"},{"why":"Supplies the silhouette coefficient used to score how tightly slide embeddings cluster by acquisition center in FM-SI.","marker":"[13]"},{"why":"Contributes the CONCH vision-language foundation model that achieves among the lowest FM-SI values.","marker":"[8]"},{"why":"Contributes the VIRCHOW-2 model, the best-performing encoder in both MIL settings in the benchmark.","marker":"[17]"}],"fun_headline_variants":["Less hospital bias, better skin cancer AI","FM-SI: the metric that spots hospital-biased pathology AI","Attention-based MIL wins for skin cancer subtyping","Center bias in features dooms skin cancer AI","New metric predicts hospital generalization of pathology AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole correlation story rests on the assumption that one t-SNE projection of mean-pooled slide embeddings faithfully captures how strongly each model's features cluster by hospital; the paper reports no projection settings or repeated runs.","fun_headline_variants_meta":{"raw":{"variants":["Less hospital bias, better skin cancer AI","FM-SI: the metric that spots hospital-biased pathology AI","Attention-based MIL wins for skin cancer subtyping","Center bias in features dooms skin cancer AI","New metric predicts hospital generalization of pathology AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001694,"raw_usage":{"total_tokens":6725,"prompt_tokens":972,"completion_tokens":5753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":5679}},"tokens_in":588,"tokens_out":5753,"duration_ms":32999,"temperature":1.0,"reasoning_tokens":5679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:44:29.127464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute FM-SI for the eight models using multiple t-SNE random seeds and perplexity settings; if the model ranking changes materially (for example, KEEP no longer has the lowest score) or the $R^2$ values against balanced accuracy in Figure 4 fall to near zero, the claimed predictive relationship would not hold.","supporting_citations":[{"cited_title":"Scientific Data12(1), 1–7 (2025)","cited_arxiv_id":null,"evidence_quote":"Provides the AI4SkIN multi-center dataset of 621 whole slide images with six cutaneous spindle cell neoplasm subtypes."}],"review_version":1}