Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A frozen 3D CT encoder's ability to detect a finding is set by the finding's contrast and spatial extent, not by the model's architecture.

desk verdict A careful, reproducible benchmark whose empirical difficulty-ordering result is credible and useful; the global-pooling causal story is a plausible inference, not a demonstrated mechanism. read the letter →

arxiv 2608.05960 v1 pith:4P5LCZZ4 submitted 2026-08-06 cs.CV cs.AI

classification cs.CVcs.AI
keywords FoundationmodelsChestCTIncidentalfindingsMulti-abnormalityclassificationFrozenfeaturesTransferevaluationContrastandextentGlobalpooling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether frozen 3D CT foundation models, used without any fine-tuning, can detect the broad range of incidental findings a radiologist would spot, and where their limits lie. Benchmarking ten encoders on three thoracic CT cohorts with k-nearest-neighbour, zero-shot, and linear-probe readouts, it finds no single best model; rankings shift by readout. The consistent result is physical: detectability scales with a finding's contrast against surrounding tissue and its spatial extent. Devices and effusions are reliably recovered, while small low-contrast focal lesions such as subsolid nodules stay near chance for every encoder, a limit the paper attributes to globally pooled embeddings.

What carries the argument

The load-bearing object is the global pooled embedding: a single vector formed from the entire scan volume, typically by aggregating patch or token features, which is what all three readouts consume. The analysis that isolates the paper's physical claim is the set of seven within-organ comparison pairs, two findings in the same organ matched to differ mainly in contrast or mainly in extent, measured by macro AUROC averaged over ten encoders. The readout decomposition (cosine k-NN, zero-shot text prompting, linear probing) shows that the difficulty hierarchy is not an artifact of a weak readout, because the linear probe extracts more signal yet still fails on focal lesions.

What would settle it

Measure the actual Hounsfield-unit contrast and volumetric extent of every finding in the seven comparison pairs and check whether AUROC differences track the measured values; alternatively, insert synthetic lesions of fixed contrast and varying size into scans and recover them with a frozen encoder. If small bright lesions are recovered as easily as large bright ones, the extent axis of the claim fails; if large low-contrast lesions are missed, the contrast axis fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a finding's detectability in a frozen 3D CT encoder is governed by two physical properties, contrast against surrounding tissue and spatial extent, rather than by model architecture or scale. This is shown with controlled within-organ comparisons: holding extent fixed while raising contrast (effusion versus thickening, solid versus subsolid nodule) improves macro AUROC by about +0.18, and holding contrast fixed while increasing extent (bulk versus faint calcification, diffuse versus focal ground-glass) improves it by about +0.16. All seven pairs show the same direction of effect with bootstrap significance, and 68 of 70 encoder-pair comparisons agree. The same easy-to-hard ordering appears across an unseen internal cohort and across k-NN, zero-shot, and linear-probe readouts, so the paper concludes that globally pooled embeddings inherently attenuate the small spatial signals of focal lesions and that those lesions will remain unrecoverable unless future pretraining uses region- or lesion-level objectives.

Load-bearing premise

The load-bearing premise is that the seven hand-picked within-organ comparison pairs isolate contrast or extent from everything else; those pairs also differ in prevalence, label-extraction pipeline, and radiological definition, so if this isolation fails, the physical-bottleneck claim reduces to saying conspicuous findings are conspicuous.

Editorial extensions

If this is right

  • If the claim holds, clinicians should not rely on frozen global CT embeddings for small low-contrast incidental findings; those findings need spatially grounded detection.
  • Benchmarking should report per-finding-type results rather than a single macro AUROC, because global scores hide the physical ceiling.
  • Pretraining objectives that operate on regions or lesions, rather than whole volumes, are the direct route to closing the focal-lesion gap.
  • Fine-grained tokenization and vision-language alignment improve retrieval and zero-shot ability but do not remove the physical bottleneck; explicit labels can substitute for scale on easy findings but not on small low-contrast ones.
  • For rare findings, the small zero-shot-over-kNN AUROC advantage vanishes under prevalence-normalized skill, so readout choice matters mainly for common abnormalities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same physical ceiling should appear in any volumetric medical-imaging encoder that pools globally, including MRI and PET, so a direct test in those modalities would show whether the mechanism generalizes.
  • One testable extension is to pretrain with a contrastive loss that operates on local patches or detected regions and compare against a global-pooling baseline on small lesions; the paper's account predicts the local objective should break the ceiling.
  • The consistency across three independently labeled cohorts suggests label noise is not the driver, but a causal version of the claim would need measured Hounsfield-unit contrast and lesion volume rather than qualitative assignment for each comparison pair.
  • The near-universal directionality across encoders implies that gain from better architecture concentrates on easy findings, so aggregate leaderboards may mislead model selection for radiology workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper benchmarks ten frozen 3D CT encoders on three thoracic CT datasets (CT-RATE, RAD-ChestCT, and an unseen internal hospital cohort) using three readouts (kNN, zero-shot prompting, and linear probing), reporting AUROC and prevalence-normalized skill. It finds no universal state-of-the-art, with fine-grained tokenization plus vision–language alignment generally leading but a lightweight supervised encoder remaining competitive. The central empirical observation is a consistent difficulty ordering across encoders and cohorts (Kendall W = 0.89): devices and fluid are easy, focal low-contrast lesions are hard. The authors attribute this to a physical bottleneck (contrast and spatial extent) and conclude that globally pooled embeddings cannot adequately represent small, low-contrast structures, recommending region- or lesion-level pretraining.

Significance. The benchmark is valuable in several respects: it evaluates a truly unseen clinical cohort, uses patient-grouped cross-validation and bootstrap confidence intervals, covers three distinct readouts, and provides code for the public cohorts. The consistent difficulty hierarchy across ten encoders and three cohorts is a robust empirical contribution that can inform model selection and evaluation design, independent of whether the causal explanation is accepted. The paper also gives credit to the possibility that explicit labels can substitute for scale, a finding that is interesting for practitioners. The weakness is the gap between the observed correlation and the causal claim about global pooling, which is not directly tested. If the causal claim is confirmed, the paper would have significant implications for pretraining objectives.

major comments (3)
  1. [Section 4, Finding Types; Fig. 4] The within-organ comparisons are presented as isolating contrast and spatial extent, but the pairs differ in multiple other dimensions beyond the claimed physical axis. Contrast and extent are assigned qualitatively, and the pairs also differ in prevalence, label-extraction pipeline, and radiological definition (e.g., pleural effusion vs. thickening; solid vs. subsolid nodule). The bootstrap-significant AUROC differences and the 68/70 directionality agreement establish a highly consistent ordering, but not that the intended physical properties are the cause. To support the causal claim, the authors should quantify contrast (e.g., HU difference to surrounding tissue) and extent (e.g., lesion volume or segmentation) for each pair, and provide a per-pair table reporting prevalence and label source, so that these potential confounds can be assessed or adjusted.
  2. [Section 3, Discussion; Abstract] The central attribution to "globally pooled embeddings" is not directly tested. No experiment compares these frozen encoders against region-level or lesion-level features under the same protocol, and no ablation removes pooling (e.g., by using patch embeddings from the same encoders instead of the global vector). The cited spatial-localization systems (Sybil, LungEvaty) are not evaluated here. As written, the conclusion is a hypothesis consistent with the data rather than a demonstrated mechanism. The authors should either add a comparison or ablation that directly addresses pooling, or rephrase the conclusion as a testable prediction, noting the absence of direct evidence.
  3. [Section 4, Datasets; Discussion, Limitations] The label-noise concern is acknowledged but dismissed based on the consistency of the difficulty ordering across independently constructed pipelines (Kendall W = 0.89). This does not rule out the confound: all three pipelines derive labels from radiology reports, whose language is systematically less reliable for subtle findings (subsolid nodules, ground-glass opacities), so the shared report-language bias could produce the same ordering even if the embeddings were insensitive to focal lesions. Provide a control using adjudicated labels or label-confidence scores (e.g., from the NLP extractors) to show that the ordering persists when label noise is reduced, or explicitly account for label uncertainty in the analysis.
minor comments (5)
  1. [Section 4, Models] The term "globally pooled embeddings" is used for all ten models, but a few models use graph/CNN pooling or other aggregation schemes. Specify the actual pooling mechanism for each encoder, or restrict the claim to models with global average pooling.
  2. [Fig. 5] Figure 5 is very dense and nearly illegible at column width; consider splitting it per cohort or enlarging the font.
  3. [Throughout] The dataset name is inconsistently written as "RadChestCT", "RAD-Chest CT", and "RAD-ChestCT"; unify the naming.
  4. [Section 2, Results] Kendall's W is reported without a confidence interval or p-value; report a bootstrap interval for the concordance statistic.
  5. [Section 2, Results] The claim that "Pillar-0 never saw CT-RATE yet remains in the top three on the two held-out cohorts" would be stronger with a statement about possible data contamination in the pretraining corpora of all models; consider discussing this.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose central claims are measured independently of their inputs, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

No circular step can be identified from the text. The paper performs an empirical evaluation of ten frozen CT encoders on three cohorts; it does not fit a parameter to a subset of data and then predict a closely related quantity. The central claim that detectability scales with contrast and spatial extent is inferred from AUROC values measured across ten encoders and multiple readouts, including an unseen internal cohort, and the empirical difficulty ordering is quantified rather than assumed. The within-organ comparisons in Fig. 4 have validity concerns—contrast and extent are assigned qualitatively, and the pairs differ in prevalence and label-extraction noise—but this is a confounding/underdetermination issue, not circularity: the AUROC values are not constructed from the physical labels. The paper also explicitly acknowledges the label-noise limitation and attempts to rebut it with Kendall W = 0.89 across independently constructed label pipelines. Self-citations appear (e.g., the RATE framework for internal-cohort labels, and Lungevaty as a spatial-localization comparison), but they are not load-bearing for the central claim, and the benchmark's result that CT-CLIP underperforms cuts against a self-serving bias. No uniqueness theorem, ansatz, or first-principles derivation is imported from the authors' prior work. The causal attribution to globally pooled embeddings is an interpretation that goes beyond the data, but that is a scientific-scope concern rather than a circular reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central benchmark claims rest on experimental choices: the model lineup, the readouts, the label extraction, and the qualitative definition of finding phenotypes. The physical-bottleneck claim additionally rests on the unmeasured assumption that the seven within-organ pairs isolate contrast and extent from confounders, and that global pooling is the causal mechanism.

free parameters (4)
  • k in kNN = 5
    Chosen by hand for the k-nearest neighbor readout; results may vary with k, though the paper states only k=5.
  • C in linear probe = 1
    Regularization strength for the l2-regularized logistic regression; fixed without sensitivity analysis.
  • Minimum positive count = 20
    Findings with fewer than 20 positives in a cohort are excluded from macro averages; this threshold affects the low-prevalence tail.
  • Within-organ comparison pairs = 7 pairs
    Hand-selected pairs for the contrast-extent analysis; the choice of pairs and the qualitative assignment of contrast and extent is a judgment call that the central claim rests on.
assumptions (4)
  • domain assumption The CT-RATE validation split is patient-disjoint from CT-RATE training data used by five models
    The paper states this but does not verify it independently (Section 4, Datasets). If near-duplicate reconstructions leaked, the CT-RATE results for those models would be inflated.
  • domain assumption NLP-extracted labels are accurate enough that label noise does not drive the focal-lesion failure
    The paper argues that consistency across three independent label pipelines (Kendall W=0.89) makes label noise unlikely (Section 3), but this is an argument, not a measurement.
  • domain assumption The within-organ pairs differ only along the intended axis (contrast or extent)
    Section 4 Finding Types and Fig. 4 rely on qualitative judgment that e.g. effusion vs. thickening is a contrast difference with extent held fixed; prevalence and other covariates are not controlled quantitatively.
  • ad hoc to paper Global average pooling is the mechanism behind the focal-lesion failure
    The paper attributes the bottleneck to globally pooled embeddings in the Abstract and Discussion, but no experiment compares against non-pooled or lesion-level models, so this is a postulated mechanism rather than a tested one.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models." pith.science (2026). https://pith.science/paper/4P5LCZZ4

@misc{pith2026260805960,
  author       = {Pith},
  title        = {Pith review of: Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4P5LCZZ4}},
  note         = {Machine review of arXiv:2608.05960}
}
abstract

Routine CT interpretation is inherently comprehensive, capturing incidental findings across the entire scan volume. 3D CT foundation models could assist this process by providing generalizable representations of anatomy and pathology. To evaluate their diagnostic breadth, we benchmark ten frozen CT encoders across three cohorts of thoracic CT scans, including an unseen internal clinical dataset, using $k$-nearest neighbors, zero-shot prompting, and linear probing. We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context. While models combining fine-grained image tokenization with vision-language alignment generally perform best, a lightweight supervised encoder remains highly competitive, demonstrating that explicit labels can effectively substitute for scale. Crucially, rather than model architecture, we observe that the primary determinant of performance is a physical bottleneck: a finding's detectability scales with its contrast against surrounding tissue and its spatial extent. Through controlled within-organ comparisons, we empirically demonstrate that widespread or high-contrast abnormalities, such as devices and effusions, are reliably recovered. Conversely, small, low-contrast focal lesions remain a persistent challenge across all evaluated encoders. We attribute this to the inherent limitations of globally pooled embeddings, suggesting that accurately representing small, low-contrast structures will require region- or lesion-level pretraining.

Figures

Figures reproduced from arXiv: 2608.05960 by the authors.

Figure 1
Figure 1. Evaluation Overview. Three 3D thoracic CT datasets are encoded us￾ing frozen foundation models. The extracted representations are then evaluated across multiple findings classification tasks using k-NN, zero-shot classification, and linear probing readouts. generalizable representations of anatomy and pathology. This generalizability is critical because medical pathologies follow a heavy-tailed distribution [20], ma… view at source ↗
Figure 2
Figure 2. Benchmarking out-of-the-box capability and data efficiency. (a) Encoder configurations and macro AUROC. Models combining vision– language alignment with fine-grained tokenizers (FlexiCT, COLIPRI) consis￾tently lead kNN and zero-shot performance across both public and unseen ex￾ternal cohorts. (b) Per-organ linear probing confirms this architectural advan￾tage persists across anatomical regions, though absolute detec… view at source ↗
Figure 3
Figure 3. Classification difficulty is dictated by finding type, not encoder. Left: Linear-probe AUROC by model and finding type. The easy-to-hard hi￾erarchy (columns, left to right) is highly consistent across all encoders: de￾vices and fluid are universally accessible, while focal lesions remain challenging. Right: Readout decomposition averaged across the six report-aligned encoders. Although linear probing extracts more l… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Contrast and spatial extent govern detectability. Controlled within-organ comparisons demonstrate that increasing either contrast (left) or spatial extent (right) improves detection. Conspicuous findings (blue) system￾atically yield higher AUROCs than their subtle coun…
Figure 5
Figure 5. Figure 5: Comprehensive performance landscape across findings, cohorts, and readouts. Per-finding AUROC for high- and low-prevalence abnormalities across three datasets, evaluated via kNN, linear probing, and zero-shot prompt￾ing. The prominent horizontal banding confirms that d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 10 canonical work pages

  1. [1]

    Research Square (2025)

    Aerts, H.J.W.L., Pai, S., Hadzic, I., Fedorov, A., Mak, R.H.: Foundation model embeddings for quantitative tumor imaging biomarkers. Research Square (2025). https://doi.org/10.21203/rs.3.rs-6630446/v1, preprint rs.3.rs-6630446

  2. [2]

    arXiv preprint arXiv:2511.17803 (2025)

    Agrawal, K.K., Liu, L., Lian, L., Nercessian, M., Harguindeguy, N., et al.: Pillar-0: a new frontier for radiology foundation models. arXiv preprint arXiv:2511.17803 (2025)

  3. [3]

    Japanese Journal of Radiology43(10), 1606– 1617 (2025)

    Aydin, S., Ece, B., Cakmak, V., Kocak, B., Onur, M.R.: Emergency radiology: roadmap for radiology departments. Japanese Journal of Radiology43(10), 1606– 1617 (2025)

  4. [4]

    arXiv preprint arXiv:2406.06512 (2024)

    Blankemeier, L., Cohen, J.P., Kumar, A., Van Veen, D., Gardezi, S.J.S., et al.: Merlin: a vision language foundation model for 3d computed tomography. arXiv preprint arXiv:2406.06512 (2024)

  5. [5]

    In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI)

    Brandt, J., Chevli, M., Braren, R., Kaissis, G., Müller, P., Rueckert, D.: Lungevaty: A scalable, open-source transformer-based deep learning model for lung cancer risk prediction in ldct screening. In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2026) 10 M. Chevli, J. Brandt et al

  6. [6]

    arXiv preprint arXiv:2511.17209 (2025)

    Claessens, C., Viviers, C., D’Amicantonio, G., Bondarev, E., van der Sommen, F.: Scaling self-supervised and cross-modal pretraining for volumetric ct transformers. arXiv preprint arXiv:2511.17209 (2025)

  7. [7]

    Structured Spectral Graph Representation Learning for Multi-label Abnormality Analysis from 3D CT Scans

    Di Piazza, T., Lazarus, C., Nempont, O., Boussel, L.: Structured spectral graph representation learning for multi-label abnormality analysis from 3d ct scans. arXiv preprint arXiv:2510.10779 (2025)

  8. [8]

    Medical Image Analysis67, 101857 (2021)

    Draelos, R.L., Dov, D., Mazurowski, M.A., Lo, J.Y., Henao, R., Rubin, G.D., Carin, L.: Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical Image Analysis67, 101857 (2021). https://doi.org/10.1016/j.media.2020.101857

Show all 20 references
  1. [9]

    Annals of emergency medicine80(3), 243–256 (2022)

    Evans, C.S., Arthur, R., Kane, M., Omofoye, F., Chung, A.E., Moreton, E., Moore, C.: Incidental radiology findings on computed tomography studies in emergency department patients: a systematic review and meta-analysis. Annals of emergency medicine80(3), 243–256 (2022)

  2. [10]

    arXiv preprint arXiv:2403.17834 (2024)

    Hamamci, I.E., Er, S., Almas, F., Simsek, A.G., Esirgun, S.N., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomog- raphy. arXiv preprint arXiv:2403.17834 (2024)

  3. [11]

    Elsevier Health Sciences (2019)

    Herring, W.: Learning radiology: recognizing the basics. Elsevier Health Sciences (2019)

  4. [12]

    Research Square (2026)

    Hoelzle, C.S., Brandt, J., Mueller, J.C., Klug, M., Westphal, J., Rueckert, D., Chevli, M., Fintelmann, F.J.: Longitudinal language-model reasoning enables au- tomated labeling of lung cancer recurrence from unstructured clinical records. Research Square (2026). https://doi.or...

  5. [13]

    arXiv preprint arXiv:2605.21906 (2026)

    Li, Y., Gao, Y., Dong, H., Lai, Y., Wang, S., Safari, M., Baciak, J.E., Yang, X.: Universal ct representations from anatomy to disease phenotype through agglom- erative pretraining. arXiv preprint arXiv:2605.21906 (2026)

  6. [14]

    Radiology312(3), e240541 (2024)

    Marcinkiewicz, A.M., Buchwald, M., Shanbhag, A., Bednarski, B.P., Killekar, A., Miller, R.J.H., Builoff, V., Lemley, M., Berman, D.S., Dey, D., Slomka, P.J.: AI for multistructure incidental findings and mortality prediction at chest CT in lung can- cer screening. Radiology312...

  7. [15]

    Journal of Clinical Oncology41(12), 2191–2200 (2023)

    Mikhael, P.G., Wohlwend, J., Yala, A., Karstens, L., Xiang, J., Takigami, A.K., Bourgouin, P.P., Chan, P., Mrah, S., Amayri, W., Juan, Y.H., Yang, C.T., Wan, Y.L., Lin, G., Sequist, L.V., Fintelmann, F.J., Barzilay, R.: Sybil: A validated deep learning model to predict future ...

  8. [16]

    arXiv preprint arXiv:2604.04133 (2026)

    Moreno-Aguado, R., Magallón, A., Moreno, V., Fang, Y., Yang, G.: Learning ro- bust visual features in computed tomography enables efficient transfer learning for clinical tasks. arXiv preprint arXiv:2604.04133 (2026)

  9. [17]

    arXiv preprint arXiv:2501.09001 (2025)

    Pai, S., et al.: Vision foundation models for computed tomography. arXiv preprint arXiv:2501.09001 (2025)

  10. [18]

    arXiv preprint arXiv:2604.01987 (2026)

    Saporta,A.,Callard,B.,Dancette,C.,Khlaut,J.,Corbière,C.,Butsanets,L.,Prat, A., Manceron, P.: Curia-2: scaling self-supervised learning for radiology foundation models. arXiv preprint arXiv:2604.01987 (2026)

  11. [19]

    arXiv preprint arXiv:2510.15042 (2025)

    Wald, T., Hamamci, I.E., Gao, Y., Bond-Taylor, S., et al.: Comprehensive language–image pre-training for 3d medical image understanding. arXiv preprint arXiv:2510.15042 (2025)

  12. [20]

    IEEE transactions on pattern analysis and machine intelligence45(9), 10795– 10816 (2023)

    Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A sur- vey. IEEE transactions on pattern analysis and machine intelligence45(9), 10795– 10816 (2023)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.