{"id":"1d12811b-3f3b-46b1-a2a7-090bbd30dc70","arxiv_id":"2508.04429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A masked autoencoder pretrained on mixed COVID-19 and ILD CT scans improves multiclass interstitial lung disease classification over supervised baselines on a small cohort.","lead":"Researchers trained a masked autoencoder on over 5,000 chest CT scans, including COVID-19 and pneumonia cases, then fine-tuned it to classify interstitial lung diseases from just 105 labeled scans. The model matched the best baseline on a binary task and outperformed it on a harder four-class task, though the gains are modest and the evaluation set is very small.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining may include the 105 fine-tuning scans; no exclusion is stated, so the reported MAE gains could reflect memorized patient identity rather than transferable features.","rationale":"The reader identified the exact load-bearing assumption: the paper never states that the 105 certified ILD scans used for fine-tuning were excluded from MAE pretraining. Section 2.2 strongly implies they are drawn from the same in-house cohort ('360 individuals in our study'), and the fine-tuning set is described separately only by the number of scans and labelling criteria, not by dataset provenance. Since the multiclass result is the primary evidence for the paper's central claim, and since the binary result is statistically tied with the baseline, the possibility of patient identity leakage is a serious correctness risk. The concern is concrete and testable: either provide patient-level disjointness or retrain with exclusion. I agree with the reader's conditional verdict; no further adjustment is needed. I would not call this a rejection because the concern may be resolvable and the paper openly provides code/models, making the retraining test feasible. The attack is not an ad hominem; it is a request for a missing data-hygiene statement plus a verification experiment.","tokens_in":8154,"tokens_out":1919,"duration_ms":22716,"concrete_test":"Ask the authors to release a de-identified patient-ID mapping or explicitly confirm whether the 105 Fleischner-certified cases are disjoint from the 360 in-house pretraining individuals. If they are not disjoint, retrain the MAE from scratch with those 105 patients excluded from pretraining, then rerun the multiclass linear probe (same 5×70/30 splits). If balanced accuracy drops to near or below the 40.8% Inception-ResNet baseline, the reported improvement is largely an artifact of pretraining on the test patients; if it remains around 49%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that MAEs trained on unlabelled CT data improve ILD classification. This requires that the 105 certified ILD scans used for fine-tuning were not also used for MAE pretraining. Section 2.2 describes pretraining on '360 individuals in our study' (in-house) and states the fine-tuning set consists of 105 certified scans from the same institution, labelled per Fleischner criteria. No sentence anywhere states that these 105 scans were excluded from the in-house pretraining set. If the same patient volumes (or even slices) appeared during reconstruction pretraining, the encoder can memorize patient-specific intensity patterns. In the binary task the MAE linear probe matches Inception-ResNet (72.3) rather than beating it, so the only strong evidence for the headline claim is the multiclass result (MAE LP 49.1% vs Inception-ResNet 40.8%, Radiomic 42.5%). An 8.3-point absolute improvement could be fully explained by pretraining on the evaluation patients. The paper's Discussion acknowledges the small evaluation dataset but does not address this separation. This is a load-bearing data-leakage risk that is internally plausible and unaddressed; it must be resolved before the multiclass superiority claim can be trusted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using a 3D masked autoencoder (MAE) pretrained on a large, largely unlabelled chest CT collection (over 5,000 scans) to learn representations for interstitial lung disease (ILD) classification. The pretrained encoder is evaluated on two downstream tasks: a binary task (UIP/probable UIP vs. indeterminate UIP/non-IPF) and a four-class multiclass task, using both full fine-tuning and linear probing. The main evidence is that the MAE linear probe achieves a multiclass balanced accuracy of 49.1% compared to 40.8% for Inception-ResNet and 42.5% for radiomics. The paper also presents ablations on patch size and a lung-aware reconstruction loss. Code and models are released.","tokens_in":8464,"tokens_out":3085,"duration_ms":30170,"significance":"If the reported gains are real, the work would be a useful demonstration that self-supervised reconstruction pretraining on heterogeneous CT data can benefit a rare-disease classification task with only 105 labelled scans. The study is honest about the small evaluation set, and it compares against several baselines, including a clinical radiomics pipeline, which is a strength. However, the central claim hinges on an unstated and load-bearing data-separation assumption: that the fine-tuning scans were not part of pretraining. This issue, together with the absence of statistical significance testing, means the headline multiclass result is not yet established. The paper is reproducible in principle (code and models are promised), but the current text does not rule out a leakage-based explanation.","major_comments":[{"comment":"The manuscript never states that the 105 certified ILD scans used for fine-tuning were excluded from MAE pretraining. The pretraining ILD data are described as coming from '360 individuals in our study,' and Table 1 reports 105 certified ILD scans from the same institution. If these 105 volumes were among the pretraining data, the encoder could memorize patient-specific intensity patterns, directly inflating the linear-probe results in Tables 4 and 5. In particular, the multiclass advantage of MAE LP (49.1%) over Inception-ResNet (40.8%) is the only strong evidence for the paper's central claim, so this ambiguity is load-bearing. Please state explicitly whether the 105 certified scans are a subset of the 360 in-house pretraining volumes, and if they are, retrain the MAE with these volumes excluded and re-report all downstream numbers.","section":"Section 2.2 (Datasets), Tables 1 and 5"},{"comment":"All results are reported as means and standard deviations over five random 70:30 splits, but no statistical significance tests, confidence intervals, or per-split results are provided. The key multiclass difference (MAE LP 49.1 ± 7.0 vs. Inception-ResNet 40.8 ± 4.7) is within one standard deviation, and with only five splits the difference could plausibly be noise. Please report paired significance tests (e.g., paired t-test or Wilcoxon across the five splits) or bootstrap confidence intervals, and include per-split balanced accuracies. Without this, the claim that the MAE 'significantly outperforms' the baseline (Discussion) is unsupported.","section":"Section 3 (Tables 4 and 5)"},{"comment":"There is a numeric inconsistency in the ILD counts. The text says the ILD-positive pretraining data consist of '360 individuals in our study' plus '176 from open-source platforms', which totals 536. Table 1, however, lists 105 certified ILD and 472 uncertified ILD, which totals 577. Please reconcile these numbers and clarify how 'certified' and 'uncertified' ILD cases relate to the fine-tuning set and to the pretraining set. This ambiguity makes it impossible to verify the data separation assumed in the evaluation.","section":"Section 2.2 and Table 1"},{"comment":"The evaluation uses scan-level 70:30 splits, but it is not stated whether the 105 certified ILD scans come from distinct patients or whether some patients contribute multiple scans. If a patient appears in both the training and validation split, identity-based shortcuts could again inflate performance. Please state whether the splits are at the patient level, and if not, rerun the evaluation with patient-level splitting.","section":"Section 2.3 (Evaluation metrics) and Section 3.1"}],"minor_comments":[{"comment":"'Acknoledgements' is a typo; should be 'Acknowledgements'.","section":"Section 5 title"},{"comment":"The caption contains a duplicated phrase: 'show our show our comparison'. Please fix.","section":"Table 4 caption"},{"comment":"The loss introduces two hyperparameters (the 25% overlap threshold and α). Only one value of α is tested (α = 0.01), and no sensitivity analysis is reported. A sentence on the choice of these values would help reproducibility.","section":"Section 3.2 (Lung-cavity-aware Reconstruction Loss)"},{"comment":"The preprocessing pipeline resamples all scans to a fixed 128×128×128 volume and applies a lung mask. It would be helpful to state the resampling target spacing explicitly, since this affects patch semantics.","section":"Section 2.2 (Preprocessing)"},{"comment":"The patch-size-8 experiment trains for 2000 epochs at batch size 8, whereas the main MAE training uses 198,000 iterations at batch size 64. The comparison is therefore confounded by compute budget. Please note this explicitly in the text.","section":"Section 3.2 (Patch Size ablation)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a clinically relevant task and provides a reasonable baseline comparison, but the data-leakage concern is serious and must be resolved before the claimed improvement can be trusted. If the authors confirm that the 105 fine-tuning scans were not used in pretraining and provide statistical significance testing, a revised version could be a solid contribution. If they find that the scans were included, the central claim would need to be substantially reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent MAE-for-medical-imaging paper with a plausible central claim, but I can't trust the headline multiclass gain until the authors confirm the 105 evaluation scans were not part of the 360 in-house pretraining volumes. That omission is not a footnote; it's load-bearing.\n\nWhat is genuinely new is the pretraining mix (COVID-19 plus ILD scans) and the lung-cavity-aware reconstruction loss that down-weights non-lung patches. Neither is a paradigm shift, but both are sensible extensions for rare-disease imaging. The paper releases code and models, and the baseline comparisons (Inception-ResNet with RadImageNet, ViT with ChexZero, and a radiomics pipeline) are reasonable for the field. The writing is clear and the ablation study is honest about what didn't help.\n\nThe soft spots are real. Section 2.2 says pretraining included 360 in-house ILD patients, and the fine-tuning set is 105 certified scans from the same institution, labeled per Fleischner criteria. Nothing in the paper states that those 105 patient volumes were excluded from pretraining. If they weren't, the MAE has seen the evaluation patients, and the reported +8.3 point multiclass improvement (49.1 vs 40.8) could be explained by memorized patient identity. The binary task shows no advantage (72.3 vs 72.3), so the multiclass result is the only evidence for the central claim. This is a testable issue—the authors can state the overlap and, if needed, retrain with the 105 excluded. But as written, the paper doesn't rule it out.\n\nA second issue is that the five 70:30 splits produce means and standard deviations but no confidence intervals or significance tests. With 74 training scans and high variance, the headline gap may be within noise even without leakage. The paper's limitation paragraph mentions the small dataset but doesn't address the pretraining overlap.\n\nTo the paper's credit, the authors openly call for an independent cohort and discuss the unstable ViT baseline. The core idea is sound; the experimental setup is under-specified. This paper belongs in a peer-review round with a clear request to verify data separation and add statistical testing. In its current form, I wouldn't cite the multiclass result as evidence that MAEs transfer better than supervised baselines.","headline":"Likely data leakage between MAE pretraining and the fine-tuning set undercuts the only strong result; the idea is worth a revision, not a rejection.","tokens_in":8967,"tokens_out":3319,"would_cite":false,"duration_ms":32789,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A masked autoencoder pretrained on over 5,000 unlabeled chest CT scans can outperform fully supervised baselines at classifying interstitial lung diseases when labelled data is scarce.","keywords":["interstitial lung disease","masked autoencoders","self-supervised learning","chest CT","medical image classification","data scarcity","linear probing","diffuse lung disease"],"falsifier":"Take the patient identifiers or file hashes of the 105 certified ILD scans and check them against the pretraining corpus. If any overlap exists, retrain the linear probe on the non-overlapping subset and see whether the 49.1% multiclass balanced accuracy falls toward the 42.5% radiomic baseline.","tokens_in":8093,"feed_emoji":"🫁","tokens_out":10609,"duration_ms":99536,"temperature":0.7,"pith_summary":"This paper claims that a masked autoencoder (MAE) pretrained on more than 5,000 unlabeled chest CT scans—mostly COVID-19, plus interstitial lung disease (ILD) and normal cases—extracts features that improve ILD diagnosis when only 105 certified scans are available for supervised training. On the harder four-way task (UIP, probable UIP, indeterminate UIP, non-IPF), a linear probe on the MAE's features reaches 49.1% balanced accuracy, versus 40.8% for a supervised Inception-ResNet baseline and 42.5% for radiomic features. On the binary fibrosis classification, the MAE linear probe ties the best supervised baseline at 72.3% balanced accuracy. The result matters because ILD imaging datasets with expert labels are small and expensive, while large unlabeled CT collections from the COVID-19 pandemic are abundant.","feed_headline":"Pretrained without labels, MAE tops ILD classification baselines","feed_subtitle":"Trained on 5,000+ unlabeled CTs, its linear probe hits 49.1% on four-way ILD diagnosis, beating supervised baselines.","key_machinery":"The engine is the masked autoencoder: a ViT-B encoder that sees only 25% of the patchified volume (128×128×128, patch size 16) and must reconstruct the 75% masked patches; the decoder is shallow and the loss is mean absolute error on the masked voxels. The forced reconstruction of invisible structure is what pushes the encoder to learn spatial and textural regularities of lung parenchyma rather than surface cues. At classification time a [CLS] token is appended to the patch sequence, and either the whole encoder is fine-tuned or only the final LayerNorm-plus-linear head is trained on class-weighted cross-entropy; the paper's key comparison is between these two readouts and supervised baselin","core_discovery":"On its own terms, the paper establishes that a masked autoencoder pretrained on CT volumes from multiple related conditions can serve as a feature extractor for interstitial lung disease classification when expert labels are scarce. The authors pretrain a ViT-B encoder on 128-cubed volumes from more than 5,000 chest CTs, using 75% random masking and voxel-level reconstruction, then evaluate two readouts on 105 certified ILD scans: full fine-tuning and linear probing on the [CLS] token. They report that linear probing gives 72.3% balanced accuracy on the binary UIP-versus-other task—tying the best supervised baseline—and 49.1% on the four-way task, beating the Inception-ResNet baseline (40.8%","pith_inferences":["If these results survive an external cohort, the same recipe—MAE pretraining on related high-volume diseases, linear probe on a small certified set—could transfer to other rare diffuse lung diseases whose CT patterns overlap with more common ones.","The consistent advantage of linear probing over full fine-tuning in both tasks suggests the pretrained features are informative but fragile; a middle path that updates only a few later layers might do even better.","Because the multiclass task requires separating overlapping UIP patterns, the MAE's larger margin there suggests reconstruction pretraining encodes textural boundaries that supervised transfer from natural-image-style weights does not capture.","A practical test: reconstruct masked lung patches and see whether UIP-specific reticulation and honeycombing are visually regenerated; if they are, reconstruction quality could double as a confidence signal for diagnosis."],"forward_implications":["Large unlabelled CT collections from related conditions such as COVID-19 and pneumonia can be substituted for expensive ILD annotations in pretraining, reducing the labelled-data barrier.","Linear probing is the recommended readout: it beat full fine-tuning in both the binary (72.3% vs 69.1%) and multiclass (49.1% vs 43.1%) tasks.","The multiclass margin over a supervised baseline (49.1% vs 40.8%) is the paper's evidence that MAE features separate overlapping UIP/non-IPF patterns better than transfer-learning baselines.","The released encoder and lung-segmentation model let other groups fine-tune for additional diffuse lung disease tasks without repeating the 5,000-scan pretraining."],"supporting_citations":[{"why":"Introduces the masked autoencoder architecture and masking-reconstruction pretraining objective that the paper adapts to 3D CT volumes.","marker":"[16]"},{"why":"Provides the 3D ViT implementation and masking recipe for medical volume pretraining.","marker":"[6]"},{"why":"Defines the supervised montage-based baseline whose binary and multiclass results the paper compares against.","marker":"[36]"},{"why":"Supplies the radiomic-feature-and-lung-pattern pipeline used as a second baseline.","marker":"[14]"},{"why":"Gives the four fibrotic-lung categories used to label the certified fine-tuning scans.","marker":"[22]"},{"why":"Documents the overlapping CT findings between COVID-19 and ILD, justifying the use of pandemic-era scans in pretraining.","marker":"[13]"},{"why":"Supplies the automatic lung segmentation used to crop every scan to the lung parenchyma before training.","marker":"[17]"},{"why":"Initializes the convolutional baseline with radiology-pretrained weights, making the comparison to transfer learning fair.","marker":"[24]"},{"why":"Initializes the ViT baseline with chest-imaging pretrained weights, serving as another supervised transfer baseline.","marker":"[34]"}],"fun_headline_variants":["MAE pretrained on 5,000 CTs beats supervised ILD baselines","Unlabeled CT pretraining lifts ILD diagnosis accuracy","Masked autoencoder pretraining tops ILD baselines","49.1% on four-way ILD: MAE beats supervised baselines","MAE with 5k CTs improves ILD four-way accuracy by 8.3 pts"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that the 105 certified ILD scans used to fine-tune and evaluate the model were not also among the unlabelled scans the MAE was pretrained on; if any of them were, the reported accuracy gains could reflect memorization of the evaluation patients rather than learned representations.","fun_headline_variants_meta":{"raw":{"variants":["MAE pretrained on 5,000 CTs beats supervised ILD baselines","Unlabeled CT pretraining lifts ILD diagnosis accuracy","Masked autoencoder pretraining tops ILD baselines","49.1% on four-way ILD: MAE beats supervised baselines","MAE with 5k CTs improves ILD four-way accuracy by 8.3 pts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000909,"raw_usage":{"total_tokens":3729,"prompt_tokens":714,"completion_tokens":3015,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2915}},"tokens_in":458,"tokens_out":3015,"duration_ms":21531,"temperature":1.0,"reasoning_tokens":2915,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:58:14.921121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the patient identifiers or file hashes of the 105 certified ILD scans and check them against the pretraining corpus. If any overlap exists, retrain the linear probe on the non-overlapping subset and see whether the 49.1% multiclass balanced accuracy falls toward the 42.5% radiomic baseline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the masked autoencoder architecture and masking-reconstruction pretraining objective that the paper adapts to 3D CT volumes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 3D ViT implementation and masking recipe for medical volume pretraining."},{"cited_title":"Lancet Respir Med (2018) 5, 6, 8","cited_arxiv_id":null,"evidence_quote":"Defines the supervised montage-based baseline whose binary and multiclass results the paper compares against."},{"cited_title":"IEEE Access (2024) 3, 4, 5, 6, 8","cited_arxiv_id":null,"evidence_quote":"Supplies the radiomic-feature-and-lung-pattern pipeline used as a second baseline."},{"cited_title":"Lancet Respiratory Medicine (2018) 2, 4","cited_arxiv_id":null,"evidence_quote":"Gives the four fibrotic-lung categories used to label the certified fine-tuning scans."},{"cited_title":"Clinical Imaging (2023) 4","cited_arxiv_id":null,"evidence_quote":"Documents the overlapping CT findings between COVID-19 and ILD, justifying the use of pandemic-era scans in pretraining."},{"cited_title":"Eur Radiol Exp (2020) 4","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic lung segmentation used to crop every scan to the lung parenchyma before training."},{"cited_title":"Radiology: Artificial Intelligence (2022) 5","cited_arxiv_id":null,"evidence_quote":"Initializes the convolutional baseline with radiology-pretrained weights, making the comparison to transfer learning fair."},{"cited_title":"Nature Biomedical Engineering (2022) 5","cited_arxiv_id":null,"evidence_quote":"Initializes the ViT baseline with chest-imaging pretrained weights, serving as another supervised transfer baseline."}],"review_version":1}