{"id":"03d352e1-a7b7-4750-8f10-c0c911e36b3a","arxiv_id":"2501.00053","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A conformalized uncertainty-aware wrapper (TRUECAM) reduces error rates and controls coverage in AI-based NSCLC subtyping on whole-slide images.","lead":"This paper introduces TRUECAM, a wrapper framework that adds uncertainty estimation, tile filtering, and statistically guaranteed error control to AI models for lung cancer subtyping on whole-slide images. It reports lower misclassification rates and better handling of out-of-distribution data across several foundation models, which matters for making AI-based cancer diagnostics safer to deploy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EAT's benefit depends on k-means centroids computed in the pre-EAT SNGP feature space, but inference uses a retrained SNGP-EAT model; the paper does not specify or validate that the ambiguous cluster transfers across these feature spaces, across datasets, or through the AutoGluon proxy.","rationale":"I agree with the reader that EAT cluster transfer is the weakest joint in the argument, but I sharpen it from an external-transfer problem to an internal representation-space mismatch. The paper's EAT procedure fits centroids to the original SNGP's tile representations, then retrains a new classifier (SNGP-EAT) on the retained tiles. At inference, it is unspecified which feature extractor produces the latent representations used for cluster assignment. If the feature extractor has changed, the centroids are in a different space and the bi-Lipschitz property of SNGP only bounds distortion relative to the original input space; it does not guarantee that the new model's representation preserves the old cluster partition. This ambiguity directly affects the central 'data trustworthiness' claim: EAT is the only component specifically responsible for removing non-informative tiles, and the strongest evidence for it is the 8.05% CPTAC improvement over SNGP. If that improvement depends on an underspecified or mismatched cluster-assignment step, the gain is not robustly established. The foundation-model version has an additional hop: an AutoGluon proxy trained on TCGA features is used not only for UNI and CONCH but also, via CONCH, for Prov-GigaPath, with no validation that ambiguity scores transfer across feature spaces or datasets. I still credit the paper's independent support: external CPTAC evaluation, a random-elimination control, reproducible code, and the fact that the conformal prediction guarantees are standard and nominally satisfied. Those supports are why the verdict should remain conditional rather than reject: the concern is specific, testable, and fixable, but until the representation-space question is resolved, the EAT contribution to the headline error-rate reductions is not fully pinned down.","tokens_in":37425,"tokens_out":5623,"duration_ms":60085,"concrete_test":"Re-run the TCGA-trained pipeline with one change: recompute k-means centroids (k=3) using SNGP-EAT's own latent representations on the retained TCGA training tiles, then apply these centroids to CPTAC inference and report patient-level accuracy, set size, and DA error rate. Compare against the paper's EAT procedure, which uses centroids from the pre-EAT SNGP. If the 8.05% CPTAC accuracy improvement over SNGP disappears or reverses, EAT's benefit depends on the representation-space mismatch, and the framework needs a stable, explicitly specified definition of the ambiguous cluster that holds for the deployed model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing component is EAT's ambiguous-tile cluster. In the specialized-model setting, k-means (k=3, chosen by Silhouette) is fitted to tile representations produced by the original SNGP on TCGA training tiles; the 66.7% ambiguous cluster is discarded and a new SNGP-EAT classifier is trained on the remaining tiles. For a new slide, the EAT section says tiles whose latent representations fall in the ambiguous cluster are excluded, but the paper does not state whether those representations come from the original SNGP or from the retrained SNGP-EAT. If the latter, the centroids live in a different feature space, so assignment to the ambiguous cluster is not guaranteed to be meaningful; if the former, the tile-selection model differs from the classification model, and the reported 8.05% CPTAC accuracy gain is not attributable to the architecture actually deployed. For foundation models the gap is wider: an AutoGluon proxy trained on TCGA tile features is used to assign ambiguity for UNI and CONCH, and the CONCH-derived AutoGluon model is reused for Prov-GigaPath, with no check that ambiguity scores order tiles by diagnostic informativeness under domain shift. If the cluster or proxy does not transfer, EAT could remove tumor-rich tiles, and the accuracy gains in Fig. 3d and Fig. 6 would be artifacts of TCGA-specific clustering rather than a general data-trustworthiness property.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces TRUECAM, a wrapper framework for NSCLC subtyping from whole-slide images that combines spectral-normalized neural Gaussian processes (SNGP) for distance-aware uncertainty estimation, an ambiguity-guided tile elimination (EAT) mechanism, and conformal prediction (CP) with conformal risk control (CRC) to provide statistical coverage guarantees and out-of-domain (OOD) detection. The authors evaluate TRUECAM on TCGA and CPTAC NSCLC cohorts using Inception-v3 and four pathology foundation models (UNI, CONCH, Prov-GigaPath, TITAN), reporting reduced error rates, improved CP efficiency, fairness gains, OOD robustness, and substantial inference speedups. The main comparisons are powered by 20 independently trained models with 500 calibration splits, and a random-elimination baseline (SNGP-RE) is included to isolate the effect of EAT.","tokens_in":37788,"tokens_out":8564,"duration_ms":80405,"significance":"If the framework's claims hold, TRUECAM would be a practically valuable model-agnostic wrapper for digital pathology AI, addressing data trustworthiness via OOD detection and ambiguous-tile filtering, and model trustworthiness via conformal prediction. The paper's strengths include the systematic use of multiple datasets, the inclusion of a random-elimination control for EAT, the breadth of foundation models considered, the reporting of statistical significance over many seeds and calibration splits, and the availability of code. The central promise—reducing misclassification while maintaining calibrated coverage—is clinically meaningful and would deserve publication once the load-bearing concerns below are resolved.","major_comments":[{"comment":"The paper does not state whether the tile representations used to assign test tiles to the ambiguous cluster at inference come from the original SNGP (whose k-means centroids are computed) or from the retrained SNGP-EAT classifier. If the latter, the centroids live in a different feature space, making cluster assignment potentially meaningless; if the former, the tile-selection model differs from the classification model, so the reported accuracy gains (Fig. 3d, 8.05% on CPTAC) cannot be attributed to the deployed architecture. For foundation models, the AutoGluon ambiguity proxy is trained on TCGA features and applied to CPTAC and, for Prov-GigaPath, to features from a different encoder (CONCH), without any explicit validation that the ambiguity ranking transfers across datasets or feature spaces. I request a precise specification of the inference-time feature extractor and a direct transferability check, such as cluster assignment consistency or an ablation using the same feature space for clustering and inference.","section":"Results: Eliminating ambiguous tiles (EAT) and Methods: Ambiguity score"},{"comment":"The claim of 'up to 1000×' inference efficiency gain without compromising accuracy is not supported by the reported data. A 1000× gain corresponds to 0.1% tile retention in Fig. 6k,l, but the accuracy at that retention appears lower than the no-elimination baseline (e.g., UNI-TRUECAM around 0.88 vs. roughly 0.92 without elimination), no statistical test is reported at that operating point, and Extended Data Fig. 9d reports slide-level inference speedups of only about 3×. Please either remove the 'up to 1000×' claim, qualify it as a tile-count reduction without demonstrating accuracy parity, or provide the corresponding benchmark and significance test.","section":"Discussion, paragraph 'EAT’s advantages are manifold'"},{"comment":"The headline error-rate reductions (72.0% and 93.8% for Inception-v3 in Fig. 1e) compare a model that can abstain (TRUECAM, output set of two labels) against a deterministic model that must always produce a single label. Because abstentions are not counted as errors, the error-rate comparison conflates deferral with improved classification. The paper does define the definitive-answer (DA) error rate in Fig. 2k,l, but the abstract and main text repeatedly state that TRUECAM 'significantly outperforms' models in classification accuracy. Please either present the DA error rate as the primary accuracy comparison or explicitly qualify the comparison as one that includes abstention.","section":"Abstract and Fig. 1e"},{"comment":"The abstract and Introduction motivate TRUECAM by addressing 'data discrepancies between model development and deployment environments,' but the foundation-model experiments train and test on the same dataset (TCGA train→TCGA test; CPTAC train→CPTAC test), so no cross-dataset distribution shift is evaluated for foundation models. The only external validation is for Inception-v3 (TCGA train→CPTAC test). This limits the generality of the 'model-agnostic' and deployment-readiness claims. I recommend either adding a cross-dataset foundation-model experiment (e.g., a TCGA-trained AutoGluon/ABMIL applied to CPTAC slides) or explicitly stating that the deployment-shift claim rests solely on the Inception-v3 experiments.","section":"Methods: Dataset configuration and Results: TRUECAM’s benefits extend to digital pathology foundation models"}],"minor_comments":[{"comment":"There are several typos, including 'Supplemtentary' (OOD detection section), 'simultanuously' (Introduction), 'their their' (Methods, SNGP), 'TURECAM' (Fig. 6h caption), and 'While' instead of 'White' (Methods, fairness evaluation).","section":"Throughout"},{"comment":"The caption reports TCGA (n=189), but the Methods define a TCGA testing set of 89 patients and a calibration set of 100; please clarify which patient set is used for the fairness analysis, as this affects the interpretation of subgroup-level gaps.","section":"Fig. 4 caption"},{"comment":"The definition of the nonconformity threshold as 'the ⌈(R+1)(1−α)⌉ / R quantile' is ambiguous; it should be stated as the ⌈(R+1)(1−α)⌉-th smallest nonconformity score in the calibration set, divided by the appropriate factor, or equivalently the empirical quantile with the standard finite-sample correction.","section":"Methods: Conformal prediction"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially valuable and the authors have done a rigorous job on the experimental side. My main concern is the EAT feature-space ambiguity and the lack of explicit transfer validation; this is fixable, but it is load-bearing for the central claim that EAT is a general data-trustworthiness mechanism. The '1000x' efficiency claim should also be reined in. I would not recommend rejection, but the revision needs to address these points convincingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ben,\n\nRead the TRUECAM paper. The headline: the Inception-v3 evaluation is the real contribution, and it's solid. They wrap a deterministic tile-level model with SNGP, an ambiguity-guided tile-removal step (EAT), and conformal prediction, then evaluate on TCGA/CPTAC with 20 independently trained models and 500 CP splits. The CPTAC external validation and the random-elimination control mean the error-rate reductions (72% and 93.8% for Inception-v3 at 0.95/0.99 coverage) are not just curve-fitting. EAT is a genuinely useful idea: k-means on SNGP tile representations to find and drop a cluster of tiles that are ambiguous for LUAD vs LUSC, then train on the rest. The pathologist-annotated sanity check for the low-ambiguity regions is a nice touch.\n\nNow the soft spots.\n\nFirst, the stress-test concern is real: the paper doesn't say whether test-tile assignments to the ambiguous cluster come from the original SNGP feature space or from the retrained SNGP-EAT model. The centroids were computed in the original space. If inference uses the retrained model's features, the cluster geometry is different, and the assignment is not guaranteed to mean anything. This needs an explicit statement and, better, an experiment that validates the cluster transfers across feature spaces, datasets, and model architectures.\n\nSecond, the foundation-model section is weaker than the Inception-v3 part. They train and evaluate UNI/CONCH on the same dataset (TCGA or CPTAC separately), so the 'model-agnostic' claim lacks external validation across cohorts. For Prov-GigaPath they reuse the CONCH-trained AutoGluon proxy without checking that ambiguity ordering transfers. That's a gap in the evidence for the general wrapper claim.\n\nThird, the 'up to 1000x efficiency' line is overstated. That figure comes from the sensitivity analysis where they retain 0.1% of tiles, not from the 60% removal used in the actual pipeline. The real speedup is closer to 2-3x. Worth correcting.\n\nFourth, the CRC/OOD calibration under varying OOD ratios is presented as if it's automatic, but the threshold choices (FPR=0.2, 0.13, etc.) are dataset-specific heuristics.\n\nNone of this kills the paper. The core empirical results for Inception-v3 are reproducible, the code and data are available, and the random-elimination baseline shows EAT isn't just removing anything. I'd send it to review, but the authors need to clarify the EAT feature-space provenance, add a genuine cross-dataset foundation-model evaluation, and tone down the efficiency claim.\n\nFor your reading group: worth discussing the EAT design, but the foundation-model claims should be taken with a grain of salt. I'd cite the Inception-v3 results if I were working on uncertainty-aware pathology AI.\n\nBest.","headline":"The Inception-v3 results are credible and well powered, but the EAT feature-space provenance and missing external validation for foundation models need work before the model-agnostic claim can be taken at face value.","tokens_in":38306,"tokens_out":3759,"would_cite":true,"duration_ms":36363,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRUECAM claims that layering SNGP uncertainty, ambiguity-guided tile elimination, and conformal prediction onto existing pathology AI models sharply reduces NSCLC subtyping errors while statistically guaranteeing coverage.","keywords":["digital pathology","non-small cell lung cancer","whole-slide images","conformal prediction","uncertainty quantification","spectral-normalized neural Gaussian process","out-of-domain detection","model fairness"],"falsifier":"On a held-out external cohort with pathologist-annotated tumor regions, measure the fraction of annotated tumor-epithelial area inside tiles that EAT discards; if that fraction is non-negligible (say above 10%), the ambiguity cluster is not transferring and the accuracy gains would not generalize.","tokens_in":37241,"feed_emoji":"🩺","tokens_out":8545,"duration_ms":81508,"temperature":0.7,"pith_summary":"TRUECAM is a wrapper framework claiming that any digital-pathology AI model for non-small cell lung cancer subtyping can be made trustworthy by stacking three mechanisms: spectral-normalized neural Gaussian process uncertainty, ambiguity-guided elimination of non-informative tiles, and conformal prediction with conformal risk control. On the paper's experiments, the wrapped Inception-v3 model misclassified 72% fewer patients at a nominal 95% coverage and 93.8% fewer at 99% coverage, while the true subtype remained in the prediction set at the promised frequency. The same wrapper improved accuracy, efficiency, out-of-domain rejection, and demographic fairness for pathology foundation models. If these claims hold, clinicians would gain a principled answer to the question 'when do I trust this AI output?': accept single-subtype predictions when the model is confident, and defer ambiguous or out-of-scope slides to a pathologist with a statistically bounded error rate.","feed_headline":"AI lung subtyping errors cut by up to 93.8 percent","feed_subtitle":"A conformal trust wrapper tells the model when to abstain, keeping true diagnoses in the predicted set at the chosen rate.","key_machinery":"Three coupled mechanisms carry the argument. Spectral-normalized neural Gaussian process (SNGP) replaces the dense output layer with a random-Fourier-feature Gaussian process and applies spectral normalization so the hidden representation preserves input distances (a bi-Lipschitz condition), making uncertainty reflect distance from training data. Elimination of ambiguous tiles (EAT) clusters SNGP tile representations on the training set (k=3 by silhouette score), identifies the cluster with no dominant subtype label, and discards it at inference, removing 66.7% of tiles in the specialized model and about 60% in foundation-model settings. Conformal prediction calibrates a nonconformity score on a separate calibration set and outputs a prediction set whose probability of containing the true subtype is at least $1-\\alpha$; conformal risk control extends this to maintain coverage when out-of-domain inputs evade detection.","core_discovery":"The paper claims that a model-agnostic trust wrapper, built from a spectral-normalized neural Gaussian process with a Gaussian process output layer, k-means-based elimination of ambiguous tiles, and conformal prediction with conformal risk control, can turn an existing NSCLC subtyping model into one with statistically guaranteed error rates. In the paper's experiments, this wrapper cut patient-level misclassification by 72% at nominal coverage 0.95 and by 93.8% at 0.99 for Inception-v3, while keeping empirical coverage close to the target and producing smaller prediction sets than Monte Carlo Dropout or deterministic baselines. It also restored coverage when out-of-domain slides were mixed into the test stream, and the same wrapper improved accuracy and fairness for pathology foundation models.","pith_inferences":["The same wrapper stack could likely transfer to other tile-based histopathology tasks such as multi-class subtyping, grading, or biomarker prediction, since SNGP and conformal prediction are task-agnostic; the paper itself only tests binary LUAD-versus-LUSC.","Because EAT's ambiguous-tile cluster is defined once on training data, a deployed system would need periodic re-derivation as scanners or staining protocols drift; the paper does not test longitudinal drift.","The fairness improvements without explicit constraints suggest that uncertainty-aware abstention can reduce subgroup gaps, but the paper's fairness analysis is limited to a few cohorts and would need broader demographic validation.","The strong results at very low tile retention rates raise the possibility of using ambiguity scores to curate pretraining data for self-supervised encoders, a direction the paper raises but does not implement."],"forward_implications":["With coverage set to $1-\\alpha=0.95$, TRUECAM cut Inception-v3's patient-level error rate by 72%; at $1-\\alpha=0.99$, the reduction was 93.8%, with empirical coverage close to the specified level.","Pairing out-of-domain detection with conformal risk control kept empirical coverage near 0.95 even when out-of-domain slides made up twice the in-domain volume, while removing both dropped coverage to 0.478.","Ambiguity-guided tile elimination improved patient-level accuracy by 2.83% on the internal test cohort and 8.05% on an external cohort, and eliminated 66.7% of tiles in the specialized model without hurting accuracy.","The same wrapper improved prediction-set efficiency and error rates for pathology foundation models, and even at a 0.1% tile retention rate ambiguity-based filtering did not degrade accuracy, unlike random filtering.","Accuracy and prediction-set-size gaps across sex and race shrank without fairness constraints in training, including race accuracy gap reductions of 38.1% and 78.3% on two cohorts."],"supporting_citations":[{"why":"Supplies the distance-aware SNGP method that TRUECAM uses for uncertainty estimation and out-of-domain detection.","marker":"[50]"},{"why":"Gives the spectral-normalized random-Fourier-feature Gaussian process formulation and bi-Lipschitz distance-preservation argument.","marker":"[56]"},{"why":"Provides the conformal prediction construction with nonconformity scores and quantile thresholds used for prediction sets.","marker":"[65]"},{"why":"Defines conformal risk control, which TRUECAM uses to maintain coverage when out-of-domain inputs escape detection.","marker":"[67]"},{"why":"Supplies the Inception-v3-based NSCLC subtyping setup and the Monte Carlo Dropout uncertainty baseline that TRUECAM is compared against.","marker":"[14]"},{"why":"Provides the UNI foundation model encoder used as a TRUECAM backend.","marker":"[26]"},{"why":"Provides the CONCH visual-language foundation model encoder used as a TRUECAM backend.","marker":"[27]"},{"why":"Provides the Prov-GigaPath whole-slide foundation model evaluated as a TRUECAM backend.","marker":"[28]"},{"why":"Provides the TITAN whole-slide foundation model evaluated as a TRUECAM backend.","marker":"[29]"},{"why":"Supplies the attention-based multiple instance learning architecture into which SNGP and EAT are inserted for foundation models.","marker":"[40]"}],"fun_headline_variants":["Conformal wrapper slashes lung AI errors by up to 93.8%","Lung AI subtyping errors cut by up to 93.8% via trust wrapper","TRUECAM ensures AI lung diagnosis with guaranteed error rates","Uncertainty-aware wrapper improves NSCLC AI accuracy","Error-controlled AI for lung cancer subtyping cuts errors 93.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cluster of ambiguous tiles identified once in the training data (one of three clusters, containing 66.7% of training tiles) remains the same in external datasets and other model architectures, so that removing it removes mostly non-informative tissue rather than diagnostic tissue.","fun_headline_variants_meta":{"raw":{"variants":["Conformal wrapper slashes lung AI errors by up to 93.8%","Lung AI subtyping errors cut by up to 93.8% via trust wrapper","TRUECAM ensures AI lung diagnosis with guaranteed error rates","Uncertainty-aware wrapper improves NSCLC AI accuracy","Error-controlled AI for lung cancer subtyping cuts errors 93.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2667,"prompt_tokens":945,"completion_tokens":1722,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1624}},"tokens_in":561,"tokens_out":1722,"duration_ms":13984,"temperature":1.0,"reasoning_tokens":1624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:42:55.731368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out external cohort with pathologist-annotated tumor regions, measure the fraction of annotated tumor-epithelial area inside tiles that EAT discards; if that fraction is non-negligible (say above 10%), the ambiguity cluster is not transferring and the accuracy gains would not generalize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the distance-aware SNGP method that TRUECAM uses for uncertainty estimation and out-of-domain detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the spectral-normalized random-Fourier-feature Gaussian process formulation and bi-Lipschitz distance-preservation argument."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Inception-v3-based NSCLC subtyping setup and the Monte Carlo Dropout uncertainty baseline that TRUECAM is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the UNI foundation model encoder used as a TRUECAM backend."},{"cited_title":"Y .et al","cited_arxiv_id":null,"evidence_quote":"Provides the CONCH visual-language foundation model encoder used as a TRUECAM backend."},{"cited_title":"& Welling, M","cited_arxiv_id":null,"evidence_quote":"Supplies the attention-based multiple instance learning architecture into which SNGP and EAT are inserted for foundation models."}],"review_version":1}