{"id":"f8885830-3dee-474a-97ed-4866a62bb8c4","arxiv_id":"2507.01881","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A lightweight 3D masked-autoencoder vision model pretrained on 98,588 low-dose CT scans achieves competitive or better AUROC than larger closed models across 14 thoracic disease classification tasks, with high label efficiency.","lead":"TANGERINE is an open-source AI model trained on 98,000 CT scans that can be fine-tuned with little data to spot lung cancer and other chest diseases in screening scans. It aims to give hospitals without big computing budgets a powerful tool for reading low-dose CT scans.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on a contaminated seen-distribution benchmark: TANGERINE was pretrained on the same 46,330 CTRate volumes later used for fine-tuning, so the reported CTRate and label-efficiency advantages are not a fair basis for 'state-of-the-art'.","rationale":"The stress-test pass confirms the reader's weakest assumption as the principal threat to the central claim. The paper is a serious empirical study with a clear architecture, large pretraining corpus, multiple downstream tasks, and useful analyses (training dynamics, frozen embeddings, uncertainty). The masked autoencoder design is plausible and the computational-frugality evidence is well documented. The problem is not the model's existence or its usefulness, but the strength of the comparative claim. The CTRate seen-distribution benchmark is the one place where the paper's own Methods text contradicts its Fig. 1 protocol statement: the exact training volumes used for fine-tuning are the same volumes used for pretraining. That is a direct violation of the stated subject-exclusion, and it selectively advantages TANGERINE over every comparator that did not pretrain on those volumes. Since the CTRate task is a multi-label 18-class benchmark used in both the headline performance figures and the label-efficiency analysis, the contamination attacks a large fraction of the evidence for 'state-of-the-art across 14 tasks' and 'high label efficiency'. The Gemini comparison asymmetry compounds rather than replaces this issue: it means the strongest external comparator is evaluated under a more restrictive protocol, so the reported margins over Gemini cannot be interpreted as fair SOTA margins. Both issues are fixable. A clean retraining with CTRate excluded from pretraining would settle the contamination question; an equal frozen-embedding comparison would settle the Gemini question. Because the central empirical claim is currently not supported as written, a conditional verdict is appropriate; this matches the reader's assessment, so no change to the verdict is recommended.","tokens_in":17678,"tokens_out":6223,"duration_ms":70147,"concrete_test":"Retrain TANGERINE on the pretraining corpus with all CTRate volumes removed (or at minimum the 46,330 CTRate training scans), then fine-tune on the CTRate training split and evaluate on the CTRate test split; recompute the CTRate results in Fig. 2a, Fig. 3a, and Extended Data Fig. 1. If the AUROC advantage over CT-Clip, ViT, and CT-Foundation-Gemini persists, the concern is refuted; if it shrinks or reverses, the headline SOTA claim is an artefact of pretraining on the fine-tuning set. As a secondary check, rerun the same benchmark with TANGERINE-Frozen (MLP on frozen embeddings) alongside Gemini-Frozen to confirm whether the end-to-end advantage survives under an equal protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that TANGERINE achieves state-of-the-art performance across 14 disease classification tasks with high label efficiency. The most load-bearing condition for that claim is that the evaluation protocol isolates the value of the pretrained representation. That condition fails on the pretrain-seen CTRate benchmark. In Methods ('Datasets for developing TANGERINE'), the authors state that the CTRate training split of 46,330 scans was 'utilised for pretraining TANGERINE', while the 3,040-scan test split was kept separate. In 'Downstream task datasets', the same 46,330 scans are used as the training set for fine-tuning on Multi-Disease-CTRate. This directly contradicts Fig. 1's 'ensuring subject exclusion' for pretrain-seen tasks. Because the fine-tuning volumes were already seen by the encoder during masked-autoencoder pretraining, TANGERINE has an unsupervised prior over the exact scans used for supervised fine-tuning; ViT, ResNet, and CT-Cancer-Foundation do not. This is not a benign distribution overlap: it inflates both the seen-distribution AUROC (Fig. 2a, Extended Data Fig. 1) and the label-efficiency results at 40% and 10% (Fig. 3a). A separate but related asymmetry is that CT-Foundation-Gemini is compared only through frozen cloud embeddings with an MLP, while TANGERINE is fine-tuned end-to-end, so the headline comparison with the closest SOTA system is not apples-to-apples. If the CTRate advantage survives a clean retraining, the concern is resolved; if not, the central SOTA claim is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TANGERINE, a 3D masked-autoencoder vision transformer (ViT-large) pretrained on 98,588 LDCT volumes from the SUMMIT study and 27 public datasets. The authors fine-tune the encoder on a range of thoracic disease classification tasks and evaluate performance in three settings: pretrain-seen distributions, pretrain-unseen distributions, and domain generalisation. They compare against several baselines including CT-Foundation-Gemini, CT-Clip, Med3D, ResNet, ViT, and CT-Cancer-Foundation, report label-efficiency results at 100%/40%/10% fine-tuning data, and analyse training dynamics, computational cost, and interpretability. The central claim is that TANGERINE achieves state-of-the-art performance across 14 disease classification tasks while being open-source, computationally frugal, and label-efficient.","tokens_in":18045,"tokens_out":5216,"duration_ms":51751,"significance":"If the reported results hold, TANGERINE would be a valuable community resource: it provides open-source code and weights, requires only four A6000 GPUs for pretraining, and demonstrates strong performance across multiple thoracic diseases and clinical centres. The empirical methodology is generally rigorous, with five random seeds, 95% confidence intervals, and Bonferroni-corrected pairwise tests; the pretraining-size ablations and the frozen-embedding results are informative. However, the headline 'state-of-the-art' claim rests on two protocol choices that appear to favour TANGERINE relative to its closest comparators, and the significance of the contribution is therefore conditional on a fair re-evaluation.","major_comments":[{"comment":"The pretrain-seen-distribution benchmark for CTRate is contaminated. The Methods state that the CTRate training set of 46,330 scans 'was utilised for pretraining TANGERINE', and the Downstream task datasets section then describes splitting this same 46,330-scan set 85:15 into training and validation for fine-tuning on Multi-Disease-CTRate. Because TANGERINE's masked-autoencoder pretraining reconstructs these exact volumes, the model has an unsupervised prior over the fine-tuning volumes; the from-scratch ViT, ResNet-50, and CT-Cancer-Foundation baselines do not share this prior (CT-Clip, which was also pretrained on CTRate, is a partial exception). This contradicts Fig. 1's assertion of 'ensuring subject exclusion' for pretrain-seen tasks and inflates the CTRate results in Fig. 2a and the label-efficiency results at 40% and 10% in Fig. 3a. Please re-run the CTRate experiments with a version of TANGERINE pretrained on the full pretraining set excluding all CTRate volumes, or equivalently fine-tune on a CTRate split whose subjects were never seen during pretraining, and report both the contaminated and clean numbers.","section":"Methods, 'Datasets for developing TANGERINE' and 'Downstream task datasets'; Fig. 1"},{"comment":"The comparison against CT-Foundation-Gemini is asymmetric. Gemini is evaluated only through frozen cloud embeddings followed by an MLP, while TANGERINE is fine-tuned end-to-end. The Results text says 'All models were fine-tuned and evaluated on the same datasets', but the Methods specify that Gemini embeddings were 'fine-tuned using the equivalent multilayer perceptron (MLP) as used for frozen TANGERINE embeddings'. Since end-to-end fine-tuning generally improves over frozen embeddings—as the authors' own TANGERINE-Frozen results demonstrate—this protocol choice can only reduce Gemini's measured performance relative to TANGERINE. Please either obtain end-to-end fine-tuned Gemini results where the API permits, or restrict the headline comparison to the frozen-embedding setting (TANGERINE-Frozen versus Gemini-Frozen) and report both comparisons. In addition, Gemini was not evaluated on any SUMMIT task, so the reported SUMMIT results are not benchmarked against the closest external model.","section":"Methods, 'Comparison models'"},{"comment":"The claim of 'state-of-the-art performance across 14 disease classification tasks' is stronger than the evidence supports. On the pretrain-unseen RadChest benchmark, TANGERINE 'matched CT-Foundation-Gemini mean AUROC across 17 classes (P=1.000)', and in the domain-generalisation setting CT-Foundation-Gemini exceeded TANGERINE on calcification (P=0.001). These are not instances of outperforming the state of the art. The abstract and conclusions should be reworded to say 'comparable or superior' performance, and the specific settings in which TANGERINE is SOTA versus comparable should be enumerated.","section":"Abstract and Results, 'Lung Disease Diagnosis'"}],"minor_comments":[{"comment":"The caption states 'Performance on CTRate classes in the pretrain-unseen-distribution setting', but the text describes evaluation on RadChest, a dataset not seen during pretraining. The caption should refer to RadChest, not CTRate.","section":"Extended Data Fig. 2 caption"},{"comment":"The sentence 'TANGERINE's ability to detect early imaging phenotypes, such as interstitial lung abnormalities (ILAs), could help could support identification and enrichment of patient cohorts' contains a duplicated phrase 'could help could support'. It should read 'could help support'.","section":"Discussion, paragraph 4"},{"comment":"The statement 'model weights are available upon request during the review process' is in tension with the paper's 'open-source' claim. Please state the exact licence and provide a public, permanent URL for the weights, or clarify the conditions under which they will be released.","section":"Data and Materials Availability"},{"comment":"The description of the CTRate split ('we split the original training set into training and validation using an 85:15 split and kept the original provided test set') should explicitly state whether the split is subject-wise and should clarify that the training/validation volumes are the same as those used for pretraining. This is important for interpreting the pretrain-seen results.","section":"Methods, 'Downstream task datasets'"},{"comment":"The sentence 'Since SUMMIT's distribution was seen during pretraining, it is considered in the pretrain-seen-distribution task dataset' is vague. Please specify whether any SUMMIT scans were used for pretraining and how the fine-tuning subsets are disjoint at the subject level from the pretraining scans.","section":"Methods, 'Datasets for developing TANGERINE'"}],"recommendation":"major_revision","confidential_remarks":"The two protocol asymmetries—CTRate pretraining overlap and the frozen-embedding-only Gemini comparison—are the main barriers to accepting the state-of-the-art claim. If the authors can provide clean CTRate results and a matched Gemini comparison, the paper is likely publishable. Regardless, the abstract's 'state-of-the-art' claim needs tempering to 'comparable or superior' in several settings. I would also recommend that the editor ask the authors to clarify the data-availability statement, since 'available upon request' is weaker than the open-source claim in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2507.01881. First, TANGERINE is a real resource: a 3D masked autoencoder pretrained on ~98K LDCT volumes (SUMMIT + 27 public sets) that can be fine-tuned on a single GPU, with code, a broad benchmark across 14 tasks, and honest compute accounting. Second, its headline claim—state-of-the-art across all 14 tasks—does not survive a close read. The CTRate seen-distribution benchmark is contaminated, and the closest comparator is not given a fair fight.\n\nWhat is actually new: the scale and heterogeneity of the pretraining corpus, the label-efficiency analysis, and the emphasis on frugality. The architecture is an incremental extension of MAE to 3D (prior work exists but isn't cited), and the paper's real contribution is the assembled corpus and the evaluation.\n\nWhat the paper does well: five seeds, confidence intervals, Bonferroni correction, multiple baselines, domain generalization, and a frozen-embedding analysis. The training dynamics plots are a nice touch. The compute reporting is transparent.\n\nThe soft spots are real but local. Methods says the CTRate 46,330-scan training split was used for pretraining, and the downstream CTRate fine-tuning set is drawn from that same split, contradicting Fig. 1's 'ensuring subject exclusion.' That inflates the seen-distribution AUROC and the label-efficiency results on CTRate. The Gemini comparison is also asymmetric: TANGERINE is fine-tuned end-to-end, while Gemini only gets an MLP on frozen embeddings. That makes the 'we beat Gemini' claims weaker than presented. These are fixable: report a clean CTRate split (or drop the task) and either fine-tune Gemini end-to-end or label the comparison as frozen-embedding-only.\n\nMinor issues: 'open-source' is qualified because weights are on request and SUMMIT is restricted; and the paper under-cites prior 3D medical MAE work.\n\nMy take: the model and corpus are worth having, and the unseen-task results (RadChest, NLST-ACRIN, domain generalization) are not contaminated. So this is not a reject; it's a major revision. The authors need to clean up the CTRate leakage and recalibrate the SOTA language. I'd send it to peer review and tell the authors to fix those two protocol issues before the claims are taken at face value.","headline":"A useful, frugal 3D MAE resource for LDCT, but the SOTA claim is overstated by a leaky CTRate seen-distribution benchmark and an unfair Gemini comparison.","tokens_in":18618,"tokens_out":3751,"would_cite":true,"duration_ms":37030,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that TANGERINE, a lightweight open-source 3D masked autoencoder pretrained on 98,588 lung CT volumes, matches or beats heavier closed models on 14 thoracic disease classification tasks while needing far less labelled…","keywords":["3D masked autoencoder","low-dose CT","lung cancer screening","self-supervised pretraining","thoracic disease classification","label efficiency","domain generalisation","open-source foundation model"],"falsifier":"Run the same 14-task benchmark with CT-Foundation-Gemini fine-tuned end-to-end on the same data splits; if it matches or exceeds TANGERINE's AUROCs, the state-of-the-art claim fails. Also, pretrain a TANGERINE variant without the 46,330 CTRate training volumes and retest on CTRate; a large drop would show the seen-distribution result depends on pretraining overlap rather than transfer.","tokens_in":17494,"feed_emoji":"🫁","tokens_out":11261,"duration_ms":112053,"temperature":0.7,"pith_summary":"This paper introduces TANGERINE, an open-source 3D masked autoencoder pretrained on 98,588 low-dose CT volumes drawn from the SUMMIT lung screening study and 27 public datasets. The central claim is that this lightweight model reaches state-of-the-art accuracy on 14 thoracic disease classification tasks, including within-one-year lung cancer, interstitial lung abnormalities, bronchiectasis, and multi-label chest disease, while generalising to unseen clinical centres and needing far less labelled data and GPU time than comparators. If true, it would let lung cancer screening programmes pivot from cancer-only reading to comprehensive respiratory disease detection using models that run locally on modest hardware, avoiding cloud dependence. The paper also reports that frozen embeddings from the pretrained encoder already support competitive classification through a shallow network.","feed_headline":"Frugal open-source CT model beats heavier rivals on 14 tasks","feed_subtitle":"Pretrained on 98,588 lung scans, TANGERINE needs a fraction of the data and GPU hours of closed rivals.","key_machinery":"The engine is a 3D masked autoencoder, a self-supervised learner that hides most of a volume and learns to reconstruct it. Volumes are resampled to $256^3$ and split into $16^3$ patches; 75% are masked, and a ViT-large encoder (24 blocks, embedding size 1,024, about 312 million parameters) processes only the visible patches, which cuts memory by roughly half and runtime by a factor of six relative to encoding all patches. A small 8-block decoder reconstructs the masked sub-volumes from the encoded visible patches and learned mask tokens, forcing the model to learn global thoracic anatomy. For downstream tasks the decoder is discarded and the encoder is fine-tuned end-to-end with a linear head, or a two-layer head for sparse cancer datasets, on a 2048-dimensional representation formed by concatenating the class token with global-average-pooled patch tokens. Frozen versions of this representation, classified by a shallow neural network, already perform competitively, and reconstructions show coherent recovery of bronchi, vessels, and pleura from heavily masked input.","core_discovery":"On the paper's own terms, the discovery is that a computationally frugal 3D masked autoencoder, in which the encoder sees only 25% of input patches during pretraining, can produce representations that match or beat much heavier and partly closed foundation models. Across pretrain-seen, pretrain-unseen, and domain-generalisation settings, TANGERINE achieves the highest mean AUROC on most tasks, with 0.894 for within-one-year cancer on SUMMIT and a statistical tie with CT-Foundation-Gemini on RadChest while surpassing it on the cancer class. With only 10% of fine-tuning data it can exceed comparators trained on 100%, and it converges in 5 to 10 epochs, cutting compute and carbon impact relative to training from scratch. The paper frames this as evidence that architectural simplicity, large-scale self-supervised pretraining, and open release are sufficient for state-of-the-art thoracic disease detection.","pith_inferences":["The paper never end-to-end fine-tunes its strongest closed competitor; until that comparison is run, the magnitude of TANGERINE's advantage over a fully adapted cloud model is untested.","On the CTRate seen-distribution task, the 46,330 fine-tuning volumes are the same scans used in pretraining, so gains there may reflect scan-level exposure rather than transferable features; an ablation that pretrains without CTRate would isolate the true transfer effect.","The paper acknowledges that $256^3$ downsampling may lose fine detail and that pretraining demographics are narrow; these caveats mean its generalisation and clinical-readiness claims await prospective multi-centre validation.","Because the authors release the 3D MAE code and weights, the same pretraining recipe could be probed on other volumetric modalities, but the paper only demonstrates thoracic CT, so cross-modality transfer remains an open conjecture."],"forward_implications":["Lung cancer screening programmes could run comprehensive multi-disease classification on-site with a single GPU, and even CPU inference for frozen embeddings, without sending scans to cloud APIs.","A centre with only 10 to 40 percent of the usual labelled data could fine-tune TANGERINE to match or beat models trained from scratch on full data, lowering annotation cost.","Models fine-tuned in one hospital should transfer to other centres and countries on tasks not seen during pretraining, reducing the need for per-site labelled retraining.","Fine-tuning reaches peak validation performance within 5 to 10 epochs, making iterative model updates for newly defined imaging phenotypes cheap in both energy and carbon.","The released weights and pipeline position the model for extension to segmentation, report generation, and quantitative scoring, not just classification."],"supporting_citations":[{"why":"Gemini-based CT foundation model is the strongest closed comparator; it defines the frozen-embedding cloud baseline TANGERINE claims to beat.","marker":"[27]"},{"why":"SUMMIT is the private screening cohort used in pretraining and the source of cancer, ILA, bronchiectasis, and nodule evaluation tasks.","marker":"[28]"},{"why":"Masked autoencoding is the self-supervised framework the paper extends to 3D volumetric LDCT.","marker":"[29]"},{"why":"Defines the Vision Transformer backbone that TANGERINE extends to 3D and that serves as a from-scratch baseline.","marker":"[30]"},{"why":"CTRate is the largest public pretraining source and the source of the CT-Clip baseline; its train/test split defines the seen-distribution benchmark.","marker":"[31]"},{"why":"Supplies the SimCLR-pretrained patch-level cancer foundation used as a comparator in both patch and full-volume forms.","marker":"[34]"},{"why":"NLST-ACRIN data provide the unseen-distribution cancer screening evaluation with 2 to 3 percent test prevalence.","marker":"[39]"},{"why":"Duke data provide a second unseen-distribution cancer screening test set.","marker":"[40]"},{"why":"RadChest provides the multi-label unseen-distribution and domain-generalisation benchmark.","marker":"[41]"}],"fun_headline_variants":["Open-source CT model matches or beats heavier rivals on 14 tasks","Frugal 3D autoencoder rivals heavy models in lung screening","TANGERINE: lightweight open-source model shines on chest CT","10% training data enough: open-source CT model matches bigger ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result assumes the comparison is fair: TANGERINE gets full end-to-end fine-tuning while its strongest closed competitor is only adapted from frozen cloud embeddings plus a small neural network, and on the CTRate benchmark the same 46,330 scans used for pretraining are also used for fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Open-source CT model matches or beats heavier rivals on 14 tasks","Frugal 3D autoencoder rivals heavy models in lung screening","TANGERINE: lightweight open-source model shines on chest CT","10% training data enough: open-source CT model matches bigger ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000601,"raw_usage":{"total_tokens":2860,"prompt_tokens":1050,"completion_tokens":1810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":1734}},"tokens_in":666,"tokens_out":1810,"duration_ms":17570,"temperature":1.0,"reasoning_tokens":1734,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:41:22.020767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 14-task benchmark with CT-Foundation-Gemini fine-tuned end-to-end on the same data splits; if it matches or exceeds TANGERINE's AUROCs, the state-of-the-art claim fails. Also, pretrain a TANGERINE variant without the 46,330 CTRate training volumes and retest on CTRate; a large drop would show the seen-distribution result depends on pretraining overlap rather than transfer.","supporting_citations":[],"review_version":1}